[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:13 dist-test-slave-9q7f krb5kdc[24417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:13 dist-test-slave-9q7f krb5kdc[24417](info): set up 2 sockets
Sep 23 17:19:13 dist-test-slave-9q7f krb5kdc[24417](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 17:19:14.036495 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:14 dist-test-slave-9q7f krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:14.164875 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40997
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:40997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:14.280440 24430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:14.280715 24430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:14.280757 24430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:14.284759 24430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:14.284827 24430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:14.284847 24430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:14.284864 24430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:14.284880 24430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:14.290270 24430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:40997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:14.291502 24430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:14.292621 24430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:19:14.301788 24430 server_base.cc:1034] running on GCE node
W20260923 17:19:14.301760 24438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:14.301846 24436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:14.302007 24435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:14.302552 24430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:14.303727 24430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:14.304919 24430 hybrid_clock.cc:648] HybridClock initialized: now 1790183954304890 us; error 42 us; skew 500 ppm
Sep 23 17:19:14 dist-test-slave-9q7f krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:14.307742 24430 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:14.308640 24430 webserver.cc:460] Webserver started at http://127.23.214.190:33667/ using document root <none> and password file <none>
I20260923 17:19:14.309161 24430 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:14.309217 24430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:14.309382 24430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:14.311463 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "32ddcace81e22a78343e29aa13d7dc60"
server_key_iv: "16bbca5b9be579fbf234cd91c00dd44a"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.311995 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "32ddcace81e22a78343e29aa13d7dc60"
server_key_iv: "16bbca5b9be579fbf234cd91c00dd44a"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.315788 24430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:14.318423 24445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:14.319705 24430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:19:14.319811 24430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "32ddcace81e22a78343e29aa13d7dc60"
server_key_iv: "16bbca5b9be579fbf234cd91c00dd44a"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.319973 24430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:14.344146 24430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:14.347286 24430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:14.347448 24430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:14.354822 24430 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:40997
I20260923 17:19:14.354841 24497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:40997 every 8 connection(s)
I20260923 17:19:14.355811 24430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:14.358944 24498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:14.362161 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 24430
I20260923 17:19:14.362344 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:14.363978 24410 external_mini_cluster.cc:1428] Setting key 18f7e0e4abc800521e14038039fdf64a
I20260923 17:19:14.365469 24498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9: Bootstrap starting.
I20260923 17:19:14.368134 24498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:14.369073 24498 log.cc:824] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:14.371469 24498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9: No bootstrap required, opened a new log
I20260923 17:19:14.375803 24498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40997 } }
I20260923 17:19:14.376081 24498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:14.376150 24498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0770c3fe99cf4e6fb1ef3c5ee1d63bd9, State: Initialized, Role: FOLLOWER
I20260923 17:19:14.376659 24498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40997 } }
I20260923 17:19:14.376829 24498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:14.376897 24498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:14.377002 24498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:14.378355 24498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40997 } }
I20260923 17:19:14.378750 24498 leader_election.cc:302] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0770c3fe99cf4e6fb1ef3c5ee1d63bd9; no voters:
I20260923 17:19:14.379141 24498 leader_election.cc:288] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:14.379262 24503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:14.379570 24503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [term 1 LEADER]: Becoming Leader. State: Replica: 0770c3fe99cf4e6fb1ef3c5ee1d63bd9, State: Running, Role: LEADER
I20260923 17:19:14.380056 24503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40997 } }
I20260923 17:19:14.380542 24498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:14.382004 24504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40997 } } }
I20260923 17:19:14.382124 24504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:14.382508 24505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0770c3fe99cf4e6fb1ef3c5ee1d63bd9. Latest consensus state: current_term: 1 leader_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770c3fe99cf4e6fb1ef3c5ee1d63bd9" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40997 } } }
I20260923 17:19:14.382602 24505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:14.382876 24515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:14.386338 24515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 17:19:14 dist-test-slave-9q7f krb5kdc[24417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:14.392859 24515 catalog_manager.cc:1347] Generated new cluster ID: 3f9fc9cb891c408e9d9aacc5838c99fc
I20260923 17:19:14.392993 24515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:14.394955 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:14.367650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.23.214.190:40997)
0923 17:19:14.368321 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:14.368342 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:14.369382 (+ 1040us) server_negotiation.cc:407] Connection header received
0923 17:19:14.375403 (+ 6021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:14.375424 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:14.375778 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:14.376128 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:14.379406 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.381092 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.383139 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.383900 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.384091 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:14.388236 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:14.388242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:14.388255 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:14.388319 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:14.390614 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:14.391278 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:14.391284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:14.391290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:14.391457 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:14.391891 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:14.391894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:14.391896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:14.392447 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:14.393276 (+ 829us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:14.393442 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 17:19:14.393739 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":150,"threads_started":1}
I20260923 17:19:14.401760 24515 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:14.402748 24515 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:14.416075 24515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0770c3fe99cf4e6fb1ef3c5ee1d63bd9: Generated new TSK 0
I20260923 17:19:14.416941 24515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:14.449991 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40997
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:14.566717 24524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:14.567010 24524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:14.567058 24524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:14.571223 24524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:14.571295 24524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:14.571357 24524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:14.577008 24524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:14.578233 24524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:14.579360 24524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:14.589538 24529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:14.589526 24530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:14.589573 24532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:14.589743 24524 server_base.cc:1034] running on GCE node
I20260923 17:19:14.590337 24524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:14.590971 24524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:14.592156 24524 hybrid_clock.cc:648] HybridClock initialized: now 1790183954592141 us; error 28 us; skew 500 ppm
Sep 23 17:19:14 dist-test-slave-9q7f krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:14.594971 24524 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:14.595883 24524 webserver.cc:460] Webserver started at http://127.23.214.129:45341/ using document root <none> and password file <none>
I20260923 17:19:14.596407 24524 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:14.596465 24524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:14.596627 24524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:14.598703 24524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "57098470ea9e4986a05eeb05ee36999d"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "65a2613d76764fda1d162e0c7ac9fe43"
server_key_iv: "ee7a1853c3fdc75f396f87ba7059edec"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.599378 24524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "57098470ea9e4986a05eeb05ee36999d"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "65a2613d76764fda1d162e0c7ac9fe43"
server_key_iv: "ee7a1853c3fdc75f396f87ba7059edec"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.603030 24524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 17:19:14.605449 24539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:14.606715 24524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:14.606812 24524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "57098470ea9e4986a05eeb05ee36999d"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "65a2613d76764fda1d162e0c7ac9fe43"
server_key_iv: "ee7a1853c3fdc75f396f87ba7059edec"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.606941 24524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:14.631325 24524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:14.634269 24524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:14.634490 24524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:14.635106 24524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:14.635954 24524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:14.636001 24524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:14.636070 24524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:14.636102 24524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:14.645190 24524 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:43909
I20260923 17:19:14.645211 24652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:43909 every 8 connection(s)
I20260923 17:19:14.646195 24524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:14.646723 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 24524
I20260923 17:19:14.646858 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:14.647128 24410 external_mini_cluster.cc:1428] Setting key 4f884b175c5c65f0373c042650e3d469
Sep 23 17:19:14 dist-test-slave-9q7f krb5kdc[24417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:14.661024 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:14.648029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:43347 (local address 127.23.214.190:40997)
0923 17:19:14.648180 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:14.648184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:14.649348 (+ 1164us) server_negotiation.cc:407] Connection header received
0923 17:19:14.650300 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:14.650303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:14.650363 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:14.650469 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:14.651659 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.652196 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.653339 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.653574 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.653690 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:14.656344 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:14.656348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:14.656350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:14.656380 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:14.658462 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:14.659117 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:14.659120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:14.659121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:14.659199 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:14.659641 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:14.659643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:14.659644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:14.659842 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:14.659916 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:14.660657 (+ 741us) server_negotiation.cc:299] Negotiation successful
0923 17:19:14.660822 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:19:14.661958 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:14.648466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40997 (local address 127.23.214.129:43347)
0923 17:19:14.649175 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:14.649228 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:14.650095 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:14.650643 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:14.650650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:14.651010 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:14.651491 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:14.651504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.652325 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.652328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:14.653209 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:14.653218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.653730 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.653734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:14.653978 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:14.654593 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:14.654608 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:14.656097 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:14.658606 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:14.658613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:14.658630 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:14.659004 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:14.659306 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:14.659313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:14.659318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:14.659532 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:14.659962 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:14.659977 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:14.660362 (+ 385us) client_negotiation.cc:769] Sending connection context
0923 17:19:14.660605 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 17:19:14.660982 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":136,"threads_started":1}
I20260923 17:19:14.663437 24653 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40997
I20260923 17:19:14.663719 24653 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:14.664265 24653 heartbeater.cc:511] Master 127.23.214.190:40997 requested a full tablet report, sending...
I20260923 17:19:14.666003 24462 ts_manager.cc:194] Registered new tserver with Master: 57098470ea9e4986a05eeb05ee36999d (127.23.214.129:43909)
I20260923 17:19:14.667307 24462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:43347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:14.677603 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40997
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:14.790132 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:14.790421 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:14.790534 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:14.794612 24658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:14.794696 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:14.794786 24658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:14.800199 24658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:14.801316 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:14.802443 24658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:14.812263 24664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:14.812263 24663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:14.812263 24666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:14.812558 24658 server_base.cc:1034] running on GCE node
I20260923 17:19:14.813098 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:14.813809 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:14.815018 24658 hybrid_clock.cc:648] HybridClock initialized: now 1790183954815001 us; error 30 us; skew 500 ppm
Sep 23 17:19:14 dist-test-slave-9q7f krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:14.817638 24658 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:14.818617 24658 webserver.cc:460] Webserver started at http://127.23.214.130:42167/ using document root <none> and password file <none>
I20260923 17:19:14.819196 24658 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:14.819289 24658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:14.819525 24658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:14.821580 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "22028e167732413e883d2b58aa0d78c4"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "864a21afd81d87cd744be85e4ecc11e4"
server_key_iv: "43e0a3e81d6b8c4d03cea7c73bb0be0a"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.822173 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "22028e167732413e883d2b58aa0d78c4"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "864a21afd81d87cd744be85e4ecc11e4"
server_key_iv: "43e0a3e81d6b8c4d03cea7c73bb0be0a"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.825834 24658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:14.828356 24673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:14.829591 24658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:14.829732 24658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "22028e167732413e883d2b58aa0d78c4"
format_stamp: "Formatted at 2026-09-23 17:19:14 on dist-test-slave-9q7f"
server_key: "864a21afd81d87cd744be85e4ecc11e4"
server_key_iv: "43e0a3e81d6b8c4d03cea7c73bb0be0a"
server_key_version: "encryptionkey@0"
I20260923 17:19:14.829865 24658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:14.841351 24658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:14.844488 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:14.844688 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:14.845266 24658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:14.846189 24658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:14.846257 24658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:14.846330 24658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:14.846383 24658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:14.855611 24658 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:40399
I20260923 17:19:14.855636 24786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:40399 every 8 connection(s)
I20260923 17:19:14.856559 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:14.864240 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 24658
I20260923 17:19:14.864387 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:14.864648 24410 external_mini_cluster.cc:1428] Setting key ac600b85f237ade75e61c27464e63bce
Sep 23 17:19:14 dist-test-slave-9q7f krb5kdc[24417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:14.871404 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:14.858542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:32901 (local address 127.23.214.190:40997)
0923 17:19:14.858698 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:14.858703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:14.859812 (+ 1109us) server_negotiation.cc:407] Connection header received
0923 17:19:14.860679 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:14.860681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:14.860731 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:14.860809 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:14.862099 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.862651 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.863935 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.864159 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.864298 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:14.867247 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:14.867254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:14.867261 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:14.867297 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:14.868887 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:14.869568 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:14.869571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:14.869573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:14.869644 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:14.870118 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:14.870121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:14.870122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:14.870358 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:14.870463 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:14.871154 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 17:19:14.871259 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:19:14.872848 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:14.858989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40997 (local address 127.23.214.130:32901)
0923 17:19:14.859647 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:14.859689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:14.860490 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:14.860995 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:14.861003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:14.861378 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:14.861942 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:14.861963 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.862805 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.862808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:14.863781 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:14.863791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:14.864371 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:14.864375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:14.864612 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:14.865316 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:14.865334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:14.867064 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:14.869008 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:14.869017 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:14.869037 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:14.869464 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:14.869745 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:14.869751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:14.869756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:14.870010 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:14.870486 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:14.870496 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:14.870873 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 17:19:14.871105 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 17:19:14.871571 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":119,"threads_started":1}
I20260923 17:19:14.874198 24787 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40997
I20260923 17:19:14.874511 24787 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:19:14.875144 24787 heartbeater.cc:511] Master 127.23.214.190:40997 requested a full tablet report, sending...
I20260923 17:19:14.876296 24462 ts_manager.cc:194] Registered new tserver with Master: 22028e167732413e883d2b58aa0d78c4 (127.23.214.130:40399)
I20260923 17:19:14.876947 24462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:32901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:14.893513 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40997
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:15.008577 24792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:15.008904 24792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:15.009017 24792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:15.013355 24792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:15.013427 24792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:15.013521 24792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:15.019120 24792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:15.020265 24792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:15.021384 24792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:15.031737 24798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:15.031728 24797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:15.031735 24800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:15.032601 24792 server_base.cc:1034] running on GCE node
I20260923 17:19:15.033186 24792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:15.034052 24792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:15.037318 24792 hybrid_clock.cc:648] HybridClock initialized: now 1790183955037274 us; error 53 us; skew 500 ppm
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183955, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:15.040524 24792 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:15.041453 24792 webserver.cc:460] Webserver started at http://127.23.214.131:46349/ using document root <none> and password file <none>
I20260923 17:19:15.042114 24792 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:15.042202 24792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:15.042452 24792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:15.044581 24792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "5e89af06e8094217b1d7e3e52e1bc015"
format_stamp: "Formatted at 2026-09-23 17:19:15 on dist-test-slave-9q7f"
server_key: "865ff177e42d79d168e935f34287847f"
server_key_iv: "a4d54f037293cb916a3e39d565f59e97"
server_key_version: "encryptionkey@0"
I20260923 17:19:15.045148 24792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "5e89af06e8094217b1d7e3e52e1bc015"
format_stamp: "Formatted at 2026-09-23 17:19:15 on dist-test-slave-9q7f"
server_key: "865ff177e42d79d168e935f34287847f"
server_key_iv: "a4d54f037293cb916a3e39d565f59e97"
server_key_version: "encryptionkey@0"
I20260923 17:19:15.049002 24792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:19:15.051591 24807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:15.052904 24792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:19:15.053049 24792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "5e89af06e8094217b1d7e3e52e1bc015"
format_stamp: "Formatted at 2026-09-23 17:19:15 on dist-test-slave-9q7f"
server_key: "865ff177e42d79d168e935f34287847f"
server_key_iv: "a4d54f037293cb916a3e39d565f59e97"
server_key_version: "encryptionkey@0"
I20260923 17:19:15.053176 24792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:15.065956 24792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:15.069325 24792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:15.069566 24792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:15.070309 24792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:15.071228 24792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:15.071326 24792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:15.071395 24792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:15.071436 24792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:15.080773 24792 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:40609
I20260923 17:19:15.080812 24920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:40609 every 8 connection(s)
I20260923 17:19:15.081836 24792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:15.090441 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 24792
I20260923 17:19:15.090569 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:15.090881 24410 external_mini_cluster.cc:1428] Setting key ac75db5dce0753fb42c31fd968adae55
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183955, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:15.097136 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:15.083936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:55657 (local address 127.23.214.190:40997)
0923 17:19:15.084084 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:15.084088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:15.085037 (+ 949us) server_negotiation.cc:407] Connection header received
0923 17:19:15.086294 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:15.086297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:15.086353 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:15.086435 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:15.087978 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.088549 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.089818 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.090143 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.090242 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:15.093441 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:15.093445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:15.093447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:15.093472 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:15.095168 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:15.095714 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:15.095716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:15.095717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:15.095791 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:15.096104 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:15.096106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:15.096107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:15.096287 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:15.096391 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:15.096891 (+ 500us) server_negotiation.cc:299] Negotiation successful
0923 17:19:15.097007 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:19:15.098342 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:15.084351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40997 (local address 127.23.214.131:55657)
0923 17:19:15.084834 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:15.084882 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:15.086080 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:15.086654 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:15.086663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:15.087074 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:15.087640 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:15.087654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.088683 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.088685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:15.089675 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:15.089685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.090365 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.090370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:15.090601 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:15.091330 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:15.091353 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:15.093195 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:15.095306 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:15.095312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:15.095326 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:15.095619 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:15.095874 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:15.095877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:15.095878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:15.096022 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:15.096391 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:15.096396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:15.096663 (+ 267us) client_negotiation.cc:769] Sending connection context
0923 17:19:15.096846 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 17:19:15.097145 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260923 17:19:15.099623 24921 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40997
I20260923 17:19:15.099967 24921 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:15.100521 24921 heartbeater.cc:511] Master 127.23.214.190:40997 requested a full tablet report, sending...
I20260923 17:19:15.101691 24462 ts_manager.cc:194] Registered new tserver with Master: 5e89af06e8094217b1d7e3e52e1bc015 (127.23.214.131:40609)
I20260923 17:19:15.102454 24462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:55657
I20260923 17:19:15.105648 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:15.132189 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:15.122609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.23.214.129:43909)
0923 17:19:15.122977 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:15.122991 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:15.123039 (+ 48us) server_negotiation.cc:407] Connection header received
0923 17:19:15.123093 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:15.123098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:15.123295 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:15.123442 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:15.124199 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.125074 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.126006 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.126393 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.126477 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:15.128929 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:15.128938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:15.128952 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:15.128976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:15.130574 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:15.130965 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:15.130970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:15.130971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:15.131046 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:15.131264 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:15.131266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:15.131267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:15.131523 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:15.131700 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:15.131908 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 17:19:15.132034 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
W20260923 17:19:15.133293 24567 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47926
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:15.159557 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:15.151158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.23.214.129:43909)
0923 17:19:15.151403 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:15.151407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:15.151424 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:19:15.151553 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:15.151556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:15.151599 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:15.151677 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:15.152372 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.152904 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.153731 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:15.154041 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:15.154127 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:15.156489 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:15.156492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:15.156494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:15.156519 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:15.158082 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:15.158479 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:15.158487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:15.158488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:15.158565 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:15.158790 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:15.158792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:15.158793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:15.158954 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:15.159050 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:15.159250 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 17:19:15.159357 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260923 17:19:15.160677 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 24524
I20260923 17:19:15.166702 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 24658
I20260923 17:19:15.174355 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 24792
I20260923 17:19:15.180263 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 24430
2026-09-23T17:19:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2231 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24935](info): set up 2 sockets
Sep 23 17:19:15 dist-test-slave-9q7f krb5kdc[24935](info): commencing operation
krb5kdc: starting...
W20260923 17:19:16.231666 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:16 dist-test-slave-9q7f krb5kdc[24935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
W20260923 17:19:17.129683 24410 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.214.190: real 0.794s user 0.000s sys 0.003s
I20260923 17:19:17.130298 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34351
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:34351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:17.245630 24948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:17.246014 24948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:17.246064 24948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:17.250450 24948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:17.250520 24948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:17.250540 24948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:17.250557 24948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:17.250573 24948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:17.255997 24948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:34351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:17.257082 24948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:17.258198 24948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:17.267597 24956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:17.267565 24954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:17.267519 24953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:17.267695 24948 server_base.cc:1034] running on GCE node
I20260923 17:19:17.268400 24948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:17.269615 24948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:17.270808 24948 hybrid_clock.cc:648] HybridClock initialized: now 1790183957270764 us; error 59 us; skew 500 ppm
Sep 23 17:19:17 dist-test-slave-9q7f krb5kdc[24935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183957, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:17.273725 24948 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:17.274767 24948 webserver.cc:460] Webserver started at http://127.23.214.190:44175/ using document root <none> and password file <none>
I20260923 17:19:17.275323 24948 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:17.275384 24948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:17.275557 24948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:17.277722 24948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "9e0fe3a824b54428bc5a0318d5af0cdc"
format_stamp: "Formatted at 2026-09-23 17:19:17 on dist-test-slave-9q7f"
server_key: "77958a8a768bc14629ddb9dcf9d55f99"
server_key_iv: "f56616a3f1eccbe0c2c8417d41b163de"
server_key_version: "encryptionkey@0"
I20260923 17:19:17.278363 24948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "9e0fe3a824b54428bc5a0318d5af0cdc"
format_stamp: "Formatted at 2026-09-23 17:19:17 on dist-test-slave-9q7f"
server_key: "77958a8a768bc14629ddb9dcf9d55f99"
server_key_iv: "f56616a3f1eccbe0c2c8417d41b163de"
server_key_version: "encryptionkey@0"
I20260923 17:19:17.282338 24948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:17.284976 24963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:17.286525 24948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:17.286700 24948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "9e0fe3a824b54428bc5a0318d5af0cdc"
format_stamp: "Formatted at 2026-09-23 17:19:17 on dist-test-slave-9q7f"
server_key: "77958a8a768bc14629ddb9dcf9d55f99"
server_key_iv: "f56616a3f1eccbe0c2c8417d41b163de"
server_key_version: "encryptionkey@0"
I20260923 17:19:17.286839 24948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:17.301837 24948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:17.305588 24948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:17.305814 24948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:17.313300 24948 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:34351
I20260923 17:19:17.313338 25015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:34351 every 8 connection(s)
I20260923 17:19:17.314383 24948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:17.316602 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 24948
I20260923 17:19:17.316704 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:17.316984 24410 external_mini_cluster.cc:1428] Setting key 5dbfa0a05ca1eb6c03f793f6d3ff75b3
I20260923 17:19:17.317600 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:17.324142 25016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc: Bootstrap starting.
Sep 23 17:19:17 dist-test-slave-9q7f krb5kdc[24935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:17.326908 25016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:17.327706 25016 log.cc:824] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:17.330210 25016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc: No bootstrap required, opened a new log
I20260923 17:19:17.332653 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:17.318536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.23.214.190:34351)
0923 17:19:17.319035 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:17.319094 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:17.319141 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:19:17.319858 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:17.319876 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:17.320226 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:17.320592 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:17.321406 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:17.322492 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:17.323364 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:17.323860 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:17.324038 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:17.326855 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:17.326861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:17.326876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:17.326910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:17.329079 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:17.329645 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:17.329650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:17.329655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:17.329990 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:17.330342 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:17.330350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:17.330352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:17.330870 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:17.331091 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:17.331421 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:19:17.331834 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":179,"threads_started":1}
I20260923 17:19:17.334941 25016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34351 } }
I20260923 17:19:17.336136 25016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:17.336236 25016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0fe3a824b54428bc5a0318d5af0cdc, State: Initialized, Role: FOLLOWER
I20260923 17:19:17.336705 25016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34351 } }
I20260923 17:19:17.336885 25016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:17.336992 25016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:17.337141 25016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:17.390244 25016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34351 } }
I20260923 17:19:17.391006 25016 leader_election.cc:302] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e0fe3a824b54428bc5a0318d5af0cdc; no voters:
I20260923 17:19:17.391489 25016 leader_election.cc:288] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:17.391642 25021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:17.391901 25021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [term 1 LEADER]: Becoming Leader. State: Replica: 9e0fe3a824b54428bc5a0318d5af0cdc, State: Running, Role: LEADER
I20260923 17:19:17.392278 25021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34351 } }
I20260923 17:19:17.393131 25016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:17.393577 25023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e0fe3a824b54428bc5a0318d5af0cdc. Latest consensus state: current_term: 1 leader_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34351 } } }
I20260923 17:19:17.393729 25023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:17.394121 25022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0fe3a824b54428bc5a0318d5af0cdc" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34351 } } }
I20260923 17:19:17.394219 25022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:17.394623 25025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:17.398099 25025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:17.403851 25025 catalog_manager.cc:1347] Generated new cluster ID: 591e8919d80f48a78249d42d8e02732a
I20260923 17:19:17.403930 25025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:17.422251 25025 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:17.423732 25025 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:17.435977 25025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e0fe3a824b54428bc5a0318d5af0cdc: Generated new TSK 0
I20260923 17:19:17.436939 25025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
W20260923 17:19:18.496562 24410 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.214.129: real 1.057s user 0.000s sys 0.003s
I20260923 17:19:18.497180 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34351
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:18.720505 25042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:18.720906 25042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:18.720996 25042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:18.726186 25042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:18.726292 25042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:18.726413 25042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:18.732990 25042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:18.734421 25042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:18.735708 25042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:19:18.747433 25042 server_base.cc:1034] running on GCE node
W20260923 17:19:18.747399 25050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:18.747411 25047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:18.747335 25048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:18.749092 25042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:18.749938 25042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:18.751153 25042 hybrid_clock.cc:648] HybridClock initialized: now 1790183958751131 us; error 38 us; skew 500 ppm
Sep 23 17:19:18 dist-test-slave-9q7f krb5kdc[24935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183958, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:18.754288 25042 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:18.755271 25042 webserver.cc:460] Webserver started at http://127.23.214.129:45815/ using document root <none> and password file <none>
I20260923 17:19:18.755893 25042 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:18.755959 25042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:18.756193 25042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:18.758541 25042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "b09c4a5154524bb5bc5977e7d07b7a16"
format_stamp: "Formatted at 2026-09-23 17:19:18 on dist-test-slave-9q7f"
server_key: "62ea50f4f3bbcaf29652adf5232beebd"
server_key_iv: "d35fab2f1fe26bd9e3a3d92229a36dc4"
server_key_version: "encryptionkey@0"
I20260923 17:19:18.759145 25042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "b09c4a5154524bb5bc5977e7d07b7a16"
format_stamp: "Formatted at 2026-09-23 17:19:18 on dist-test-slave-9q7f"
server_key: "62ea50f4f3bbcaf29652adf5232beebd"
server_key_iv: "d35fab2f1fe26bd9e3a3d92229a36dc4"
server_key_version: "encryptionkey@0"
I20260923 17:19:18.763113 25042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:19:18.765815 25057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:18.767481 25042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:18.767645 25042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b09c4a5154524bb5bc5977e7d07b7a16"
format_stamp: "Formatted at 2026-09-23 17:19:18 on dist-test-slave-9q7f"
server_key: "62ea50f4f3bbcaf29652adf5232beebd"
server_key_iv: "d35fab2f1fe26bd9e3a3d92229a36dc4"
server_key_version: "encryptionkey@0"
I20260923 17:19:18.767782 25042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:18.783272 25042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:18.786636 25042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:18.786859 25042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:18.787520 25042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:18.788378 25042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:18.788429 25042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:18.788470 25042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:18.788530 25042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:18.799743 25042 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:39085
I20260923 17:19:18.799763 25170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:39085 every 8 connection(s)
I20260923 17:19:18.800730 25042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:18.806205 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 25042
I20260923 17:19:18.806322 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:18.806620 24410 external_mini_cluster.cc:1428] Setting key 48c07aded991e0d8bc7887df0901c497
Sep 23 17:19:18 dist-test-slave-9q7f krb5kdc[24935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183958, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:18.816336 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:18.802722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:59043 (local address 127.23.214.190:34351)
0923 17:19:18.802980 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:18.802984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:18.803658 (+ 674us) server_negotiation.cc:407] Connection header received
0923 17:19:18.805087 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:18.805090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:18.805156 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:18.805316 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:18.806880 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:18.807536 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:18.808763 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:18.808993 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:18.809105 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:18.811960 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:18.811966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:18.811971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:18.812015 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:18.814009 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:18.814580 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:18.814587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:18.814589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:18.814697 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:18.815050 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:18.815056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:18.815061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:18.815323 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:18.815436 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:18.815963 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 17:19:18.816113 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":109,"threads_started":1}
I20260923 17:19:18.817116 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:18.803080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34351 (local address 127.23.214.129:59043)
0923 17:19:18.803506 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:18.803548 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:18.804854 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:18.805502 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:18.805510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:18.805998 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:18.806715 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:18.806737 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:18.807667 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:18.807670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:18.808630 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:18.808639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:18.809158 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:18.809163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:18.809352 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:18.810053 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:18.810070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:18.811742 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:18.814139 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:18.814144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:18.814157 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:18.814468 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:18.814805 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:18.814807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:18.814809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:18.814949 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:18.815436 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:18.815441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:18.815704 (+ 263us) client_negotiation.cc:769] Sending connection context
0923 17:19:18.815877 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 17:19:18.816159 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:19:18.818538 25171 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34351
I20260923 17:19:18.818809 25171 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:18.819350 25171 heartbeater.cc:511] Master 127.23.214.190:34351 requested a full tablet report, sending...
I20260923 17:19:18.820861 24980 ts_manager.cc:194] Registered new tserver with Master: b09c4a5154524bb5bc5977e7d07b7a16 (127.23.214.129:39085)
I20260923 17:19:18.822038 24980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:59043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
W20260923 17:19:19.809190 24410 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.214.130: real 1.001s user 0.000s sys 0.003s
I20260923 17:19:19.809729 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34351
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:19:19.824952 25171 heartbeater.cc:503] Master 127.23.214.190:34351 was elected leader, sending a full tablet report...
W20260923 17:19:19.930160 25177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:19.930455 25177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:19.930501 25177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:19.934667 25177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:19.934744 25177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:19.934842 25177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:19.940388 25177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:19.941617 25177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:19.942700 25177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:19.953013 25182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:19.953008 25183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:19.953351 25177 server_base.cc:1034] running on GCE node
W20260923 17:19:19.953510 25185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:19.954046 25177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:19.954753 25177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:19.955940 25177 hybrid_clock.cc:648] HybridClock initialized: now 1790183959955905 us; error 49 us; skew 500 ppm
Sep 23 17:19:19 dist-test-slave-9q7f krb5kdc[24935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183959, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:19.959189 25177 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:19.960165 25177 webserver.cc:460] Webserver started at http://127.23.214.130:35071/ using document root <none> and password file <none>
I20260923 17:19:19.960731 25177 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:19.960795 25177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:19.960965 25177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:19.963207 25177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "b9abb638c3794be69e5de19133fd0604"
format_stamp: "Formatted at 2026-09-23 17:19:19 on dist-test-slave-9q7f"
server_key: "6aab8e75fdee8a03ba914c22e4a70f36"
server_key_iv: "32946eeff1d9ac8c5071b97e74122e85"
server_key_version: "encryptionkey@0"
I20260923 17:19:19.963771 25177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "b9abb638c3794be69e5de19133fd0604"
format_stamp: "Formatted at 2026-09-23 17:19:19 on dist-test-slave-9q7f"
server_key: "6aab8e75fdee8a03ba914c22e4a70f36"
server_key_iv: "32946eeff1d9ac8c5071b97e74122e85"
server_key_version: "encryptionkey@0"
I20260923 17:19:19.967964 25177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:19:19.970760 25192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:19.972632 25177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:19:19.972792 25177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "b9abb638c3794be69e5de19133fd0604"
format_stamp: "Formatted at 2026-09-23 17:19:19 on dist-test-slave-9q7f"
server_key: "6aab8e75fdee8a03ba914c22e4a70f36"
server_key_iv: "32946eeff1d9ac8c5071b97e74122e85"
server_key_version: "encryptionkey@0"
I20260923 17:19:19.972922 25177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:20.011691 25177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:20.015717 25177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:20.015942 25177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:20.016822 25177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:20.017846 25177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:20.017969 25177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 17:19:20.018013 25177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:20.018034 25177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:20.027226 25177 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:36481
I20260923 17:19:20.027289 25305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:36481 every 8 connection(s)
I20260923 17:19:20.028185 25177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:20.037290 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 25177
I20260923 17:19:20.037451 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:20.037735 24410 external_mini_cluster.cc:1428] Setting key 4081a45fd7c4a02990bb6608ce8d251c
Sep 23 17:19:20 dist-test-slave-9q7f krb5kdc[24935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183959, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:20.043097 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:20.030081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:39815 (local address 127.23.214.190:34351)
0923 17:19:20.030326 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:20.030331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:20.031162 (+ 831us) server_negotiation.cc:407] Connection header received
0923 17:19:20.032547 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:20.032551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:20.032612 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:20.032728 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:20.034036 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.034595 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.035825 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.036063 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.036181 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:20.039068 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:20.039073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:20.039078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:20.039107 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:20.040741 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:20.041316 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:20.041319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:20.041321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:20.041402 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:20.041782 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:20.041785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:20.041787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:20.042043 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:20.042155 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:20.042844 (+ 689us) server_negotiation.cc:299] Negotiation successful
0923 17:19:20.042961 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260923 17:19:20.044400 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:20.030497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34351 (local address 127.23.214.130:39815)
0923 17:19:20.030960 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:20.031006 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:20.032298 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:20.032915 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:20.032922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:20.033296 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:20.033809 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:20.033823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.034743 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.034746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:20.035658 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:20.035668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.036222 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.036226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:20.036408 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:20.037025 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:20.037041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:20.038889 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:20.040882 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:20.040890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:20.040908 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:20.041188 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:20.041510 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:20.041514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:20.041516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:20.041674 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:20.042153 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:20.042160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:20.042538 (+ 378us) client_negotiation.cc:769] Sending connection context
0923 17:19:20.042771 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 17:19:20.043174 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260923 17:19:20.045758 25306 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34351
I20260923 17:19:20.046036 25306 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:20.046545 25306 heartbeater.cc:511] Master 127.23.214.190:34351 requested a full tablet report, sending...
I20260923 17:19:20.047573 24980 ts_manager.cc:194] Registered new tserver with Master: b9abb638c3794be69e5de19133fd0604 (127.23.214.130:36481)
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:19:20.048179 24980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:39815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
W20260923 17:19:20.751237 24410 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.214.131: real 0.712s user 0.002s sys 0.002s
I20260923 17:19:20.751742 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34351
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:20.867682 25312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:20.868022 25312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:20.868108 25312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:20.872566 25312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:20.872654 25312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:20.872749 25312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:20.878541 25312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:20.879794 25312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:20.880955 25312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:20.891379 25320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:20.891575 25312 server_base.cc:1034] running on GCE node
W20260923 17:19:20.891390 25317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:20.891407 25318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:20.892251 25312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:20.893016 25312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:20.894248 25312 hybrid_clock.cc:648] HybridClock initialized: now 1790183960894237 us; error 32 us; skew 500 ppm
Sep 23 17:19:20 dist-test-slave-9q7f krb5kdc[24935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183960, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:20.897289 25312 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:20.898443 25312 webserver.cc:460] Webserver started at http://127.23.214.131:39341/ using document root <none> and password file <none>
I20260923 17:19:20.899045 25312 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:20.899140 25312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:20.899394 25312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:20.901569 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "ece23f88a8cf474cbf2784d67641458d"
format_stamp: "Formatted at 2026-09-23 17:19:20 on dist-test-slave-9q7f"
server_key: "566ba28cabbe6f64cdb9046a8dc36a70"
server_key_iv: "ab9f2eb25ccb5c10f0b2a62240efd93d"
server_key_version: "encryptionkey@0"
I20260923 17:19:20.902211 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "ece23f88a8cf474cbf2784d67641458d"
format_stamp: "Formatted at 2026-09-23 17:19:20 on dist-test-slave-9q7f"
server_key: "566ba28cabbe6f64cdb9046a8dc36a70"
server_key_iv: "ab9f2eb25ccb5c10f0b2a62240efd93d"
server_key_version: "encryptionkey@0"
I20260923 17:19:20.906375 25312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:19:20.909035 25327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:20.910573 25312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:20.910720 25312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "ece23f88a8cf474cbf2784d67641458d"
format_stamp: "Formatted at 2026-09-23 17:19:20 on dist-test-slave-9q7f"
server_key: "566ba28cabbe6f64cdb9046a8dc36a70"
server_key_iv: "ab9f2eb25ccb5c10f0b2a62240efd93d"
server_key_version: "encryptionkey@0"
I20260923 17:19:20.910846 25312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:20.924592 25312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:20.928234 25312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:20.928458 25312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:20.929095 25312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:20.930051 25312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:20.930119 25312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:20.930186 25312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:20.930236 25312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:20.940714 25312 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:46145
I20260923 17:19:20.940780 25440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:46145 every 8 connection(s)
I20260923 17:19:20.941757 25312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:20.948587 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 25312
I20260923 17:19:20.948730 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:20.949044 24410 external_mini_cluster.cc:1428] Setting key 7c4188a68194454ee7932e40a7e9405a
Sep 23 17:19:20 dist-test-slave-9q7f krb5kdc[24935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183960, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:20.958865 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:20.944667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:59861 (local address 127.23.214.190:34351)
0923 17:19:20.944947 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:20.944951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:20.945659 (+ 708us) server_negotiation.cc:407] Connection header received
0923 17:19:20.947186 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:20.947190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:20.947266 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:20.947390 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:20.948783 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.949578 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.951427 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.951702 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.951789 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:20.954882 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:20.954885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:20.954887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:20.954912 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:20.956713 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:20.957237 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:20.957239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:20.957241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:20.957347 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:20.957655 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:20.957657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:20.957658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:20.957850 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:20.958008 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:20.958606 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 17:19:20.958724 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
I20260923 17:19:20.959661 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:20.944991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34351 (local address 127.23.214.131:59861)
0923 17:19:20.945488 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:20.945535 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:20.946929 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:20.947599 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:20.947607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:20.948017 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:20.948576 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:20.948592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.949769 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.949774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:20.951289 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:20.951302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.951878 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.951882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:20.952053 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:20.952764 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:20.952782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:20.954695 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:20.956832 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:20.956837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:20.956853 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:20.957131 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:20.957427 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:20.957429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:20.957431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:20.957571 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:20.958028 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:20.958033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:20.958287 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 17:19:20.958461 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 17:19:20.958741 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":125,"threads_started":1}
I20260923 17:19:20.960932 25441 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34351
I20260923 17:19:20.961325 25441 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:20.962031 25441 heartbeater.cc:511] Master 127.23.214.190:34351 requested a full tablet report, sending...
I20260923 17:19:20.963207 24979 ts_manager.cc:194] Registered new tserver with Master: ece23f88a8cf474cbf2784d67641458d (127.23.214.131:46145)
I20260923 17:19:20.963892 24979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:59861
I20260923 17:19:20.975798 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:19:20.988411 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:20.979878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.23.214.190:34351)
0923 17:19:20.980014 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:20.980018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:20.980246 (+ 228us) server_negotiation.cc:407] Connection header received
0923 17:19:20.980382 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:20.980384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:20.980431 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:20.980510 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:20.981192 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.981738 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.982725 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:20.982966 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:20.983046 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:20.984435 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:20.984438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:20.984440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:20.984465 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:20.986178 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:20.986585 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:20.986588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:20.986589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:20.986666 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:20.987615 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:20.987618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:20.987619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:20.987853 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:20.987964 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:20.988157 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:19:20.988260 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:19:20.993940 24979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:20.997339 24979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:21.013100 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.007802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:36481 (local address 127.0.0.1:44608)
0923 17:19:21.008279 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:21.008292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:21.008367 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:21.009094 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:21.009100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:21.009121 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:21.009207 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.009217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.010574 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.010579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.011699 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.011711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.012754 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.012758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.012841 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.012860 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:19:21.012917 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 17:19:21.012966 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":78,"threads_started":1}
I20260923 17:19:21.013229 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.008213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.23.214.130:36481)
0923 17:19:21.008623 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.008629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.008651 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:19:21.008727 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.008734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.008925 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.009070 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:21.009333 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.010461 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.011824 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.012638 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.012771 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.012900 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.012958 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.013054 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":132,"threads_started":1}
I20260923 17:19:21.013403 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.007200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:39085 (local address 127.0.0.1:50784)
0923 17:19:21.007863 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:21.007918 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:21.008042 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:21.008918 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:21.008921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:21.008938 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:21.009049 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.009058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.010169 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.010175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.011584 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.011602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.012980 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.012984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.013068 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.013090 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:19:21.013160 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 17:19:21.013255 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":11,"thread_start_us":77,"threads_started":1}
I20260923 17:19:21.013656 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.007124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.23.214.131:46145)
0923 17:19:21.007625 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.007638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.008003 (+ 365us) server_negotiation.cc:407] Connection header received
0923 17:19:21.008240 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.008247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.008488 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.008615 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:21.009086 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.010325 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.011724 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.012366 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.012512 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.012664 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.013165 (+ 501us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.013503 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":158,"threads_started":1}
I20260923 17:19:21.013906 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.007399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.23.214.129:39085)
0923 17:19:21.007962 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.007971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.007997 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:19:21.008525 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.008530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.008776 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.008946 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:21.009177 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.010064 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.011787 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.012638 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.012771 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.012900 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.013633 (+ 733us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.013739 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":104,"threads_started":1}
I20260923 17:19:21.013100 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.006985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:46145 (local address 127.0.0.1:38828)
0923 17:19:21.007861 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:21.007916 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:21.008077 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:21.008786 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:21.008792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:21.008815 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:21.008953 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.008960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.010463 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.010467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.011584 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.011602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.012495 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.012498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.012604 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.012638 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:19:21.012761 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:19:21.012918 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":125,"threads_started":1}
I20260923 17:19:21.016909 25240 tablet_service.cc:1467] Processing CreateTablet for tablet e05205798dea47c58cf7f953f42b97d9 (DEFAULT_TABLE table=test-table [id=225a53fc92f4460291a620d54a36799f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:21.016906 25375 tablet_service.cc:1467] Processing CreateTablet for tablet e05205798dea47c58cf7f953f42b97d9 (DEFAULT_TABLE table=test-table [id=225a53fc92f4460291a620d54a36799f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:21.017998 25240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05205798dea47c58cf7f953f42b97d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:21.018004 25375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05205798dea47c58cf7f953f42b97d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:21.020113 25105 tablet_service.cc:1467] Processing CreateTablet for tablet e05205798dea47c58cf7f953f42b97d9 (DEFAULT_TABLE table=test-table [id=225a53fc92f4460291a620d54a36799f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:21.021204 25105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05205798dea47c58cf7f953f42b97d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:21.024705 25460 tablet_bootstrap.cc:492] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604: Bootstrap starting.
I20260923 17:19:21.026197 25461 tablet_bootstrap.cc:492] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d: Bootstrap starting.
I20260923 17:19:21.027275 25460 tablet_bootstrap.cc:654] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:21.028231 25460 log.cc:824] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:21.029023 25462 tablet_bootstrap.cc:492] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16: Bootstrap starting.
I20260923 17:19:21.031406 25462 tablet_bootstrap.cc:654] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:21.032362 25462 log.cc:824] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:21.032429 25460 tablet_bootstrap.cc:492] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604: No bootstrap required, opened a new log
I20260923 17:19:21.032833 25460 ts_tablet_manager.cc:1397] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:19:21.034536 25462 tablet_bootstrap.cc:492] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16: No bootstrap required, opened a new log
I20260923 17:19:21.034729 25462 ts_tablet_manager.cc:1397] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:19:21.037037 25460 raft_consensus.cc:348] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.037309 25460 raft_consensus.cc:374] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:21.037361 25460 raft_consensus.cc:729] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9abb638c3794be69e5de19133fd0604, State: Initialized, Role: FOLLOWER
I20260923 17:19:21.037915 25460 consensus_queue.cc:260] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.039041 25460 ts_tablet_manager.cc:1428] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:19:21.039036 25462 raft_consensus.cc:348] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.039310 25462 raft_consensus.cc:374] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:21.039358 25462 raft_consensus.cc:729] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09c4a5154524bb5bc5977e7d07b7a16, State: Initialized, Role: FOLLOWER
I20260923 17:19:21.039408 25306 heartbeater.cc:503] Master 127.23.214.190:34351 was elected leader, sending a full tablet report...
I20260923 17:19:21.039918 25462 consensus_queue.cc:260] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.040900 25462 ts_tablet_manager.cc:1428] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:19:21.046509 25461 tablet_bootstrap.cc:654] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:21.048183 25461 log.cc:824] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:21.050781 25461 tablet_bootstrap.cc:492] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d: No bootstrap required, opened a new log
I20260923 17:19:21.051076 25461 ts_tablet_manager.cc:1397] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.005s
W20260923 17:19:21.053339 25172 tablet.cc:2369] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:21.055040 25461 raft_consensus.cc:348] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.055333 25461 raft_consensus.cc:374] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:21.055387 25461 raft_consensus.cc:729] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece23f88a8cf474cbf2784d67641458d, State: Initialized, Role: FOLLOWER
I20260923 17:19:21.055892 25461 consensus_queue.cc:260] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.056632 25441 heartbeater.cc:503] Master 127.23.214.190:34351 was elected leader, sending a full tablet report...
I20260923 17:19:21.058151 25461 ts_tablet_manager.cc:1428] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:19:21.072072 25468 raft_consensus.cc:482] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:21.072257 25468 raft_consensus.cc:504] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.073479 25468 leader_election.cc:288] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9abb638c3794be69e5de19133fd0604 (127.23.214.130:36481), b09c4a5154524bb5bc5977e7d07b7a16 (127.23.214.129:39085)
I20260923 17:19:21.078276 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.073801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:50355 (local address 127.23.214.130:36481)
0923 17:19:21.073988 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.073993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.074010 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:19:21.074065 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.074069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.074119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.074214 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:21.074521 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.075310 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.076925 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.077565 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.077729 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.077799 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.078053 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.078134 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260923 17:19:21.078267 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.073649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:36481 (local address 127.23.214.131:50355)
0923 17:19:21.073786 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:21.073803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:21.073927 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:21.074246 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:21.074252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:21.074275 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:21.074404 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.074410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.075423 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.075425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.076778 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.076804 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.077719 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.077722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.077868 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.077939 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 17:19:21.078022 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:19:21.078084 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":47}
I20260923 17:19:21.078440 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.074035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:56215 (local address 127.23.214.129:39085)
0923 17:19:21.074155 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.074161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.074320 (+ 159us) server_negotiation.cc:407] Connection header received
0923 17:19:21.074401 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.074403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.074440 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.074535 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:21.074752 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.075464 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.076893 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.077626 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.077729 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.077799 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.078231 (+ 432us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.078323 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:19:21.078456 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.073754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:39085 (local address 127.23.214.131:56215)
0923 17:19:21.074220 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:21.074234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:21.074324 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:21.074553 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:21.074555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:21.074569 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:21.074652 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.074656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.075551 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.075554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.076761 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:21.076772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.077937 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.077941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:21.078073 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.078097 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:19:21.078206 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:19:21.078269 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":23,"thread_start_us":166,"threads_started":1}
I20260923 17:19:21.079092 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05205798dea47c58cf7f953f42b97d9" candidate_uuid: "ece23f88a8cf474cbf2784d67641458d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9abb638c3794be69e5de19133fd0604" is_pre_election: true
I20260923 17:19:21.079090 25125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05205798dea47c58cf7f953f42b97d9" candidate_uuid: "ece23f88a8cf474cbf2784d67641458d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" is_pre_election: true
I20260923 17:19:21.079421 25260 raft_consensus.cc:2454] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece23f88a8cf474cbf2784d67641458d in term 0.
I20260923 17:19:21.079420 25125 raft_consensus.cc:2454] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece23f88a8cf474cbf2784d67641458d in term 0.
I20260923 17:19:21.079972 25330 leader_election.cc:302] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b09c4a5154524bb5bc5977e7d07b7a16, b9abb638c3794be69e5de19133fd0604, ece23f88a8cf474cbf2784d67641458d; no voters:
I20260923 17:19:21.080281 25468 raft_consensus.cc:2781] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:21.080376 25468 raft_consensus.cc:482] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:21.080441 25468 raft_consensus.cc:3037] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:21.127993 25468 raft_consensus.cc:504] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.128582 25468 leader_election.cc:288] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [CANDIDATE]: Term 1 election: Requested vote from peers b9abb638c3794be69e5de19133fd0604 (127.23.214.130:36481), b09c4a5154524bb5bc5977e7d07b7a16 (127.23.214.129:39085)
I20260923 17:19:21.129119 25260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05205798dea47c58cf7f953f42b97d9" candidate_uuid: "ece23f88a8cf474cbf2784d67641458d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9abb638c3794be69e5de19133fd0604"
I20260923 17:19:21.129154 25125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05205798dea47c58cf7f953f42b97d9" candidate_uuid: "ece23f88a8cf474cbf2784d67641458d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09c4a5154524bb5bc5977e7d07b7a16"
I20260923 17:19:21.129298 25260 raft_consensus.cc:3037] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:21.129372 25125 raft_consensus.cc:3037] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:21.130626 25260 raft_consensus.cc:2454] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece23f88a8cf474cbf2784d67641458d in term 1.
I20260923 17:19:21.131120 25331 leader_election.cc:302] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9abb638c3794be69e5de19133fd0604, ece23f88a8cf474cbf2784d67641458d; no voters:
I20260923 17:19:21.131319 25125 raft_consensus.cc:2454] T e05205798dea47c58cf7f953f42b97d9 P b09c4a5154524bb5bc5977e7d07b7a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece23f88a8cf474cbf2784d67641458d in term 1.
I20260923 17:19:21.131454 25468 raft_consensus.cc:2781] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:21.131709 25468 raft_consensus.cc:686] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [term 1 LEADER]: Becoming Leader. State: Replica: ece23f88a8cf474cbf2784d67641458d, State: Running, Role: LEADER
I20260923 17:19:21.132135 25468 consensus_queue.cc:237] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } }
I20260923 17:19:21.135443 24979 catalog_manager.cc:5495] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d reported cstate change: term changed from 0 to 1, leader changed from <none> to ece23f88a8cf474cbf2784d67641458d (127.23.214.131). New cstate: current_term: 1 leader_uuid: "ece23f88a8cf474cbf2784d67641458d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece23f88a8cf474cbf2784d67641458d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 46145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9abb638c3794be69e5de19133fd0604" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b09c4a5154524bb5bc5977e7d07b7a16" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 39085 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[24935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:21.149757 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.140779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.23.214.129:39085)
0923 17:19:21.140928 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.140932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.141156 (+ 224us) server_negotiation.cc:407] Connection header received
0923 17:19:21.141251 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.141254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.141299 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.141385 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:21.142125 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.142638 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.143444 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.143649 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.143724 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.146354 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:21.146368 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:21.146384 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:21.146409 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:21.148164 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:21.148567 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:21.148572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:21.148574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:21.148653 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:21.148889 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:21.148891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:21.148892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:21.149141 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:21.149295 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.149453 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.149551 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[24935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260923 17:19:21.192823 25442 tablet.cc:2369] T e05205798dea47c58cf7f953f42b97d9 P ece23f88a8cf474cbf2784d67641458d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[24935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:21.239506 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.229349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.23.214.129:39085)
0923 17:19:21.229538 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.229545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.229688 (+ 143us) server_negotiation.cc:407] Connection header received
0923 17:19:21.229817 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.229822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.229917 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.230041 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:21.230752 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.231491 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.232377 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.232608 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.232721 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.235391 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:21.235394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:21.235396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:21.235421 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:21.236955 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:21.238116 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:21.238121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:21.238123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:21.238218 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:21.238514 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:21.238519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:21.238521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:21.238751 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:21.238888 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.239194 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.239339 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[24935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260923 17:19:21.279911 25307 tablet.cc:2369] T e05205798dea47c58cf7f953f42b97d9 P b9abb638c3794be69e5de19133fd0604: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[24935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:21.343616 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:21.330627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.23.214.129:39085)
0923 17:19:21.330824 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:21.330832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:21.331110 (+ 278us) server_negotiation.cc:407] Connection header received
0923 17:19:21.331260 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:21.331267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:21.331339 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:21.331443 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:21.332204 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.332993 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.333845 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:21.334096 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:21.334183 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:21.336758 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:21.336761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:21.336763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:21.336791 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:21.341533 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:21.342182 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:21.342186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:21.342189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:21.342298 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:21.342615 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:21.342619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:21.342620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:21.342894 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:21.343045 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:21.343303 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 17:19:21.343438 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260923 17:19:21.345680 25085 tablet_service.cc:2908] Rejecting scan request for tablet e05205798dea47c58cf7f953f42b97d9: Uninitialized: clean time has not yet been initialized
I20260923 17:19:21.348526 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 25042
I20260923 17:19:21.354653 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 25177
I20260923 17:19:21.361218 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 25312
I20260923 17:19:21.367614 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 24948
2026-09-23T17:19:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6187 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[25484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[25484](info): set up 2 sockets
Sep 23 17:19:21 dist-test-slave-9q7f krb5kdc[25484](info): commencing operation
krb5kdc: starting...
W20260923 17:19:22.892287 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.500s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:22 dist-test-slave-9q7f krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:23.023018 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:43953
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:43953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:23.144503 25497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:23.144810 25497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:23.144855 25497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:23.149035 25497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:23.149111 25497 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:19:23.149129 25497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:23.149147 25497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:23.149164 25497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:23.149216 25497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:23.154948 25497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:43953
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:43953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:23.156126 25497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:23.157227 25497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:23.166237 25502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:23.166232 25503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:23.166229 25505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:23.166640 25497 server_base.cc:1034] running on GCE node
I20260923 17:19:23.167248 25497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:23.168447 25497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:23.169641 25497 hybrid_clock.cc:648] HybridClock initialized: now 1790183963169621 us; error 34 us; skew 500 ppm
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:23.172948 25497 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:23.174006 25497 webserver.cc:460] Webserver started at http://127.23.214.190:39575/ using document root <none> and password file <none>
I20260923 17:19:23.174592 25497 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:23.174659 25497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:23.174851 25497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:23.177078 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "fb7aaafa704c4491bf96737634d66eb5"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "8fc0c45ee274ee7506c6dada4fbab984"
server_key_iv: "5ebd0621bc640dcebc484babfcbc99fc"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.177726 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "fb7aaafa704c4491bf96737634d66eb5"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "8fc0c45ee274ee7506c6dada4fbab984"
server_key_iv: "5ebd0621bc640dcebc484babfcbc99fc"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.182070 25497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:23.184644 25512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:23.186055 25497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:23.186201 25497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "fb7aaafa704c4491bf96737634d66eb5"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "8fc0c45ee274ee7506c6dada4fbab984"
server_key_iv: "5ebd0621bc640dcebc484babfcbc99fc"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.186331 25497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:23.201416 25497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:23.205327 25497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:23.205559 25497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:23.213728 25497 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:43953
I20260923 17:19:23.213778 25574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:43953 every 8 connection(s)
I20260923 17:19:23.214743 25497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:23.217628 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:23.220727 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 25497
I20260923 17:19:23.220860 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:23.221120 24410 external_mini_cluster.cc:1428] Setting key a5eaee74c85ec45f2cecf0f0659093ae
I20260923 17:19:23.224261 25575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5: Bootstrap starting.
I20260923 17:19:23.227165 25575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:23.228046 25575 log.cc:824] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:23.230332 25575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5: No bootstrap required, opened a new log
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:23.235093 25575 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7aaafa704c4491bf96737634d66eb5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43953 } }
I20260923 17:19:23.235419 25575 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:23.235477 25575 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7aaafa704c4491bf96737634d66eb5, State: Initialized, Role: FOLLOWER
I20260923 17:19:23.236032 25575 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7aaafa704c4491bf96737634d66eb5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43953 } }
I20260923 17:19:23.236192 25575 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:23.236266 25575 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:23.236377 25575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:23.237365 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.222532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.23.214.190:43953)
0923 17:19:23.222986 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:23.223006 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:23.223050 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:19:23.223873 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:23.223896 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:23.224322 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:23.224785 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:23.225428 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.227025 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.228131 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.228752 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.228886 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.231583 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:23.231590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:23.231608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:23.231656 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:23.233851 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.234359 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.234366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.234372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.234546 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.234849 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.234852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.234854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.235437 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:23.235681 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:23.235992 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 17:19:23.236310 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":119,"threads_started":1}
I20260923 17:19:23.237560 25575 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7aaafa704c4491bf96737634d66eb5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43953 } }
I20260923 17:19:23.237936 25575 leader_election.cc:302] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb7aaafa704c4491bf96737634d66eb5; no voters:
I20260923 17:19:23.238346 25575 leader_election.cc:288] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:23.238442 25580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:23.238602 25580 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [term 1 LEADER]: Becoming Leader. State: Replica: fb7aaafa704c4491bf96737634d66eb5, State: Running, Role: LEADER
I20260923 17:19:23.238936 25580 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7aaafa704c4491bf96737634d66eb5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43953 } }
I20260923 17:19:23.239670 25575 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:23.240888 25582 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb7aaafa704c4491bf96737634d66eb5. Latest consensus state: current_term: 1 leader_uuid: "fb7aaafa704c4491bf96737634d66eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7aaafa704c4491bf96737634d66eb5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43953 } } }
I20260923 17:19:23.240911 25581 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb7aaafa704c4491bf96737634d66eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7aaafa704c4491bf96737634d66eb5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43953 } } }
I20260923 17:19:23.241017 25582 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:23.241020 25581 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:23.241434 25589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:23.244930 25589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:23.250946 25589 catalog_manager.cc:1347] Generated new cluster ID: 19dc3cebf398415baf579c3c6d8e51fe
I20260923 17:19:23.251044 25589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:23.267710 25589 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:23.269150 25589 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:23.273608 25589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5: Generated new TSK 0
I20260923 17:19:23.274523 25589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:23.317327 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43953
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:23.435142 25601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:23.435482 25601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:23.435570 25601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:23.439723 25601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:23.439841 25601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:23.439914 25601 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:23.439992 25601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:23.445703 25601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:23.446926 25601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:23.448019 25601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:23.458086 25609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:23.458084 25607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:23.458086 25606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:23.458468 25601 server_base.cc:1034] running on GCE node
I20260923 17:19:23.458881 25601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:23.459585 25601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:23.460796 25601 hybrid_clock.cc:648] HybridClock initialized: now 1790183963460763 us; error 59 us; skew 500 ppm
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:23.463768 25601 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:23.464690 25601 webserver.cc:460] Webserver started at http://127.23.214.129:42225/ using document root <none> and password file <none>
I20260923 17:19:23.465219 25601 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:23.465291 25601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:23.465456 25601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:23.467620 25601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "f421ed4943264178960a1c07ed0f14b1"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "18d6e6db0baaace54d939e1fe0a6f2d3"
server_key_iv: "0b4c1e7e9913734a93243d64dc149ebf"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.468189 25601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "f421ed4943264178960a1c07ed0f14b1"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "18d6e6db0baaace54d939e1fe0a6f2d3"
server_key_iv: "0b4c1e7e9913734a93243d64dc149ebf"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.472151 25601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:23.474748 25616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:23.476084 25601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:23.476188 25601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "f421ed4943264178960a1c07ed0f14b1"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "18d6e6db0baaace54d939e1fe0a6f2d3"
server_key_iv: "0b4c1e7e9913734a93243d64dc149ebf"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.476367 25601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:23.499563 25601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:23.503147 25601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:23.503381 25601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:23.504971 25601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:23.505023 25601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:23.505088 25601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:23.505121 25601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:23.516777 25601 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:44363
I20260923 17:19:23.516816 25731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:44363 every 8 connection(s)
I20260923 17:19:23.518086 25601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:23.520136 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.505829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:42099 (local address 127.23.214.190:43953)
0923 17:19:23.506006 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:23.506011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:23.507052 (+ 1041us) server_negotiation.cc:407] Connection header received
0923 17:19:23.508384 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:23.508388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:23.508450 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:23.508548 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:23.510527 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.511173 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.512568 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.512823 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.512940 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.516089 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:23.516092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:23.516094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:23.516121 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:23.517832 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.518467 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.518474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.518479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.518585 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.519000 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.519003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.519005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.519199 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:23.519316 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:23.519825 (+ 509us) server_negotiation.cc:299] Negotiation successful
0923 17:19:23.519966 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:19:23.520900 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.506348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43953 (local address 127.23.214.129:42099)
0923 17:19:23.506860 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:23.506926 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:23.508130 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:23.508876 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:23.508888 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:23.509397 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:23.510339 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.510364 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.511302 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.511305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.512440 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.512450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.512996 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.513000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.513184 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.513943 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:23.513970 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:23.515905 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:23.517975 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.517981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.517997 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.518344 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.518700 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.518705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.518708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.518887 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.519313 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:23.519320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:23.519583 (+ 263us) client_negotiation.cc:769] Sending connection context
0923 17:19:23.519772 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 17:19:23.520096 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260923 17:19:23.522161 25732 heartbeater.cc:347] Connected to a master server at 127.23.214.190:43953
I20260923 17:19:23.522324 25732 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:23.522806 25732 heartbeater.cc:511] Master 127.23.214.190:43953 requested a full tablet report, sending...
I20260923 17:19:23.524526 25524 ts_manager.cc:194] Registered new tserver with Master: f421ed4943264178960a1c07ed0f14b1 (127.23.214.129:44363)
I20260923 17:19:23.525188 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 25601
I20260923 17:19:23.525309 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:23.525574 24410 external_mini_cluster.cc:1428] Setting key 32fcccf1218086cf67b9b435ca8cd8f9
I20260923 17:19:23.526167 25524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:42099
I20260923 17:19:23.532840 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.524242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43953 (local address 127.23.214.129:36191)
0923 17:19:23.524556 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:23.524571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:23.524675 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:23.524910 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:23.524913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:23.525124 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:23.525251 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.525259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.526349 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.526354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.527113 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.527123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.527781 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.527785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.527892 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.528487 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:23.528506 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:23.529009 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:23.531168 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.531171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.531172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.531391 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.531732 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.531734 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.531736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.531820 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.532295 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:23.532297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:23.532378 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 17:19:23.532466 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:19:23.532645 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260923 17:19:23.533102 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.524319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:36191 (local address 127.23.214.190:43953)
0923 17:19:23.524438 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:23.524441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:23.524691 (+ 250us) server_negotiation.cc:407] Connection header received
0923 17:19:23.524761 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:23.524763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:23.524799 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:23.524865 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:23.525386 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.526212 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.527231 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.527669 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.527761 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.529143 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:23.529146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:23.529148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:23.529169 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:23.531048 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.531510 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.531517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.531522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.531618 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.531914 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.531920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.531925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.532198 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:23.532309 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:23.532562 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 17:19:23.532707 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:23.557044 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43953
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:23.668974 25741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:23.669241 25741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:23.669327 25741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:23.673624 25741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:23.673697 25741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:23.673772 25741 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:23.673859 25741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:23.679373 25741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:23.680567 25741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:23.681636 25741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:23.691448 25749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:23.691423 25747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:23.691557 25741 server_base.cc:1034] running on GCE node
W20260923 17:19:23.691740 25746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:23.692142 25741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:23.692893 25741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:23.694068 25741 hybrid_clock.cc:648] HybridClock initialized: now 1790183963694054 us; error 24 us; skew 500 ppm
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:23.701713 25741 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:23.702769 25741 webserver.cc:460] Webserver started at http://127.23.214.130:44579/ using document root <none> and password file <none>
I20260923 17:19:23.703375 25741 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:23.703442 25741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:23.703608 25741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:23.705632 25741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "0b5e922a8dca47dcb40e355a2cf54428"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "5fca4b843bd6cb18186325f80e3d3c9c"
server_key_iv: "83d9c2407fe5ecdeb84260ecd909cd4f"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.706194 25741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "0b5e922a8dca47dcb40e355a2cf54428"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "5fca4b843bd6cb18186325f80e3d3c9c"
server_key_iv: "83d9c2407fe5ecdeb84260ecd909cd4f"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.709791 25741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:19:23.712399 25756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:23.713726 25741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:23.713835 25741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "0b5e922a8dca47dcb40e355a2cf54428"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "5fca4b843bd6cb18186325f80e3d3c9c"
server_key_iv: "83d9c2407fe5ecdeb84260ecd909cd4f"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.714000 25741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:23.744015 25741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:23.747438 25741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:23.747592 25741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:23.748965 25741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:23.749043 25741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:23.749116 25741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:23.749166 25741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:23.760708 25741 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:40199
I20260923 17:19:23.760744 25871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:40199 every 8 connection(s)
I20260923 17:19:23.761929 25741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:23.763535 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.750011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:50019 (local address 127.23.214.190:43953)
0923 17:19:23.750143 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:23.750146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:23.751233 (+ 1087us) server_negotiation.cc:407] Connection header received
0923 17:19:23.752447 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:23.752450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:23.752506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:23.752614 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:23.754243 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.754869 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.756096 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.756392 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.756511 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.759500 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:23.759504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:23.759506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:23.759543 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:23.761245 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.761796 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.761803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.761808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.761934 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.762373 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.762377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.762378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.762617 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:23.762718 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:23.763301 (+ 583us) server_negotiation.cc:299] Negotiation successful
0923 17:19:23.763415 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:19:23.763983 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 25741
I20260923 17:19:23.764111 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:23.764407 24410 external_mini_cluster.cc:1428] Setting key 75e061ae11fce13232490fd2241716b6
I20260923 17:19:23.764551 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.750574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43953 (local address 127.23.214.130:50019)
0923 17:19:23.751066 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:23.751108 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:23.752194 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:23.752788 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:23.752797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:23.753292 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:23.754084 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.754103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.755010 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.755014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.755947 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.755956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.756564 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.756569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.756764 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.757437 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:23.757454 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:23.759303 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:23.761391 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.761397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.761409 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.761669 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.762053 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.762057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.762059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.762226 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.762773 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:23.762779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:23.763028 (+ 249us) client_negotiation.cc:769] Sending connection context
0923 17:19:23.763186 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 17:19:23.763456 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260923 17:19:23.765949 25872 heartbeater.cc:347] Connected to a master server at 127.23.214.190:43953
I20260923 17:19:23.766134 25872 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:23.766672 25872 heartbeater.cc:511] Master 127.23.214.190:43953 requested a full tablet report, sending...
I20260923 17:19:23.767866 25524 ts_manager.cc:194] Registered new tserver with Master: 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130:40199)
I20260923 17:19:23.768723 25524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:50019
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:19:23.776836 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.768422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43953 (local address 127.23.214.130:52567)
0923 17:19:23.768702 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:23.768723 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:23.768799 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:23.769430 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:23.769434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:23.769661 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:23.769774 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.769781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.770958 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.770962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.771581 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.771590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.772268 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.772271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.772346 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.772950 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:23.772961 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:23.773314 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:23.775320 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.775324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.775326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.775532 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.775839 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:23.775841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:23.775843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:23.775922 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:23.776401 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:23.776403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:23.776484 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 17:19:23.776573 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:19:23.776690 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":122,"threads_started":1}
I20260923 17:19:23.776865 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.769064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:52567 (local address 127.23.214.190:43953)
0923 17:19:23.769190 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:23.769197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:23.769213 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:19:23.769251 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:23.769254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:23.769298 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:23.769419 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:23.769943 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.770817 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.771709 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.772129 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.772256 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.773470 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:23.773476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:23.773481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:23.773512 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:23.775196 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.775658 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.775661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.775662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.775747 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:23.776003 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:23.776005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:23.776006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:23.776317 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:23.776399 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:23.776635 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 17:19:23.776733 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:23.796940 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43953
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:23.913338 25881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:23.913674 25881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:23.913758 25881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:23.918135 25881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:23.918237 25881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:23.918313 25881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:23.918393 25881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:23.924116 25881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:23.925278 25881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:23.926402 25881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:23.936455 25886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:23.936458 25889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:23.936458 25887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:23.936872 25881 server_base.cc:1034] running on GCE node
I20260923 17:19:23.937312 25881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:23.938093 25881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:23.939277 25881 hybrid_clock.cc:648] HybridClock initialized: now 1790183963939258 us; error 36 us; skew 500 ppm
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:23.942397 25881 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:23.943410 25881 webserver.cc:460] Webserver started at http://127.23.214.131:40679/ using document root <none> and password file <none>
I20260923 17:19:23.944059 25881 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:23.944161 25881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:23.944428 25881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:23.946758 25881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "f66cdff20193405f8363abd92e5b2e98"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "3563b0d9d1a29001b180784dffd47d60"
server_key_iv: "0fb3045238c5a576a2d53732c51edf7e"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.947392 25881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "f66cdff20193405f8363abd92e5b2e98"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "3563b0d9d1a29001b180784dffd47d60"
server_key_iv: "0fb3045238c5a576a2d53732c51edf7e"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.951803 25881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:23.954691 25896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:23.956094 25881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:19:23.956259 25881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "f66cdff20193405f8363abd92e5b2e98"
format_stamp: "Formatted at 2026-09-23 17:19:23 on dist-test-slave-9q7f"
server_key: "3563b0d9d1a29001b180784dffd47d60"
server_key_iv: "0fb3045238c5a576a2d53732c51edf7e"
server_key_version: "encryptionkey@0"
I20260923 17:19:23.956401 25881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:23.984413 25881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:23.987622 25881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:23.987846 25881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:23.989401 25881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:23.989514 25881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:23.989609 25881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:23.989662 25881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:23 dist-test-slave-9q7f krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:24.000838 25881 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:35621
I20260923 17:19:24.000898 26011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:35621 every 8 connection(s)
I20260923 17:19:24.001856 25881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:24.003902 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 25881
I20260923 17:19:24.004030 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:24.004365 24410 external_mini_cluster.cc:1428] Setting key 1f499af3fb88ba2b9baa5267d5fe574a
I20260923 17:19:24.005414 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.990281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:40887 (local address 127.23.214.190:43953)
0923 17:19:23.990416 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:23.990419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:23.991870 (+ 1451us) server_negotiation.cc:407] Connection header received
0923 17:19:23.993119 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:23.993123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:23.993245 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:23.993379 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:23.995165 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.995703 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.997747 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.998035 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.998121 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.001370 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:24.001374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:24.001376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:24.001402 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:24.003179 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.003750 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.003753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.003755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.003829 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.004228 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.004231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.004232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.004461 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:24.004548 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.005086 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.005231 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:19:24.006394 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:23.991109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43953 (local address 127.23.214.131:40887)
0923 17:19:23.991699 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:23.991752 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:23.992877 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:23.993599 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:23.993610 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:23.994173 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:23.994969 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.994993 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.995860 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.995868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.997604 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:23.997616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:23.998193 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:23.998198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:23.998419 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:23.999216 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:23.999244 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:24.001172 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:24.003308 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:24.003315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:24.003333 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:24.003604 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:24.003925 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:24.003928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:24.003930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:24.004133 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:24.004558 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:24.004564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:24.004843 (+ 279us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.005048 (+ 205us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.005401 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":125,"threads_started":1}
I20260923 17:19:24.007556 26012 heartbeater.cc:347] Connected to a master server at 127.23.214.190:43953
I20260923 17:19:24.007716 26012 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:24.008217 26012 heartbeater.cc:511] Master 127.23.214.190:43953 requested a full tablet report, sending...
I20260923 17:19:24.009280 25524 ts_manager.cc:194] Registered new tserver with Master: f66cdff20193405f8363abd92e5b2e98 (127.23.214.131:35621)
I20260923 17:19:24.010105 25524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:40887
I20260923 17:19:24.016397 26018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.009648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43953 (local address 127.23.214.131:53065)
0923 17:19:24.010023 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.010038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.010136 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.010471 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.010475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.010687 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:24.010807 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.010813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.011839 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.011841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.012399 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.012407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.012839 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.012843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.012948 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.013400 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:24.013411 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:24.013714 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:24.015217 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:24.015220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:24.015221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:24.015409 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:24.015639 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:24.015641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:24.015642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:24.015710 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:24.016010 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:24.016012 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:24.016080 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.016156 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.016257 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260923 17:19:24.016587 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.009786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:53065 (local address 127.23.214.190:43953)
0923 17:19:24.010007 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.010010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.010105 (+ 95us) server_negotiation.cc:407] Connection header received
0923 17:19:24.010278 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.010281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.010356 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.010440 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:24.010945 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.011699 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.012515 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.012725 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.012816 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.013835 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:24.013838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:24.013840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:24.013860 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:24.015123 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.015491 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.015493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.015494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.015563 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.015782 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.015784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.015785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.015936 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:24.016009 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.016288 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.016385 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260923 17:19:24.018555 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:19:24 dist-test-slave-9q7f krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183964, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:19:24 dist-test-slave-9q7f krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183964, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:24.044757 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.036184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.23.214.190:43953)
0923 17:19:24.036339 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.036343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.036356 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:19:24.036401 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.036404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.036442 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.036555 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:24.037270 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.037809 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.038716 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.038943 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.039047 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.041960 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:24.041963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:24.041965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:24.041993 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:24.043219 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.043601 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.043604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.043605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.043681 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.043916 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.043918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.043919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.044089 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:24.044187 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.044469 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.044566 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:19:24.047911 25524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:24.051018 25524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:24.065330 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.059409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:44363 (local address 127.0.0.1:36806)
0923 17:19:24.059814 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.059873 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.060037 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.060744 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.060750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.060776 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.060972 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.060980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.062499 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.062506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.063742 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.063751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.064542 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.064545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.064863 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.064915 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.065046 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.065169 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":31,"thread_start_us":114,"threads_started":1}
I20260923 17:19:24.065320 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.059566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.23.214.129:44363)
0923 17:19:24.059809 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.059816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.059965 (+ 149us) server_negotiation.cc:407] Connection header received
0923 17:19:24.060232 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.060256 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.060559 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.061140 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.061279 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.062398 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.063862 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.064419 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.064555 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.064752 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.065077 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.065178 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":56,"threads_started":1}
I20260923 17:19:24.065666 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.060238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:35621 (local address 127.0.0.1:57142)
0923 17:19:24.060764 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.060780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.060870 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.061589 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.061591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.061606 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.061687 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.061692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.063329 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.063332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.064100 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.064109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.065269 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.065273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.065359 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.065387 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.065433 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.065495 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":19,"thread_start_us":95,"threads_started":1}
I20260923 17:19:24.066236 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.060687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.23.214.131:35621)
0923 17:19:24.061092 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.061100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.061117 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:19:24.061216 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.061223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.061431 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.061617 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.061777 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.063193 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.064238 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.065136 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.065812 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.065963 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.066014 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.066098 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":119,"threads_started":1}
I20260923 17:19:24.067344 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.060571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40199 (local address 127.0.0.1:48826)
0923 17:19:24.061363 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.061376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.061468 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.061932 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.061935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.061952 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.062041 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.062047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.063332 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.063334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.065493 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.065501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.066940 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.066944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.067027 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.067048 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.067128 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.067192 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":89,"thread_start_us":106,"threads_started":1}
I20260923 17:19:24.067394 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.060990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.23.214.130:40199)
0923 17:19:24.061336 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.061347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.061464 (+ 117us) server_negotiation.cc:407] Connection header received
0923 17:19:24.061577 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.061582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.061762 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.061960 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.062136 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.063210 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.065660 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.066580 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.066753 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.066877 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.067176 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.067264 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
I20260923 17:19:24.068903 25666 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda5bc0635442789e34ca2a15153e1f (DEFAULT_TABLE table=test-table [id=acb2b59e8f0f4d088bdd10c400d5fb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:24.069097 25946 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda5bc0635442789e34ca2a15153e1f (DEFAULT_TABLE table=test-table [id=acb2b59e8f0f4d088bdd10c400d5fb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:24.070071 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda5bc0635442789e34ca2a15153e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.070078 25946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda5bc0635442789e34ca2a15153e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.070032 25806 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda5bc0635442789e34ca2a15153e1f (DEFAULT_TABLE table=test-table [id=acb2b59e8f0f4d088bdd10c400d5fb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:24.070976 25806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda5bc0635442789e34ca2a15153e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.076994 26035 tablet_bootstrap.cc:492] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Bootstrap starting.
I20260923 17:19:24.077926 26036 tablet_bootstrap.cc:492] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Bootstrap starting.
I20260923 17:19:24.078356 26037 tablet_bootstrap.cc:492] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Bootstrap starting.
I20260923 17:19:24.079370 26035 tablet_bootstrap.cc:654] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.080196 26037 tablet_bootstrap.cc:654] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.080327 26036 tablet_bootstrap.cc:654] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.080338 26035 log.cc:824] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:24.080964 26037 log.cc:824] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:24.081313 26036 log.cc:824] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:24.082556 26035 tablet_bootstrap.cc:492] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: No bootstrap required, opened a new log
I20260923 17:19:24.082780 26035 ts_tablet_manager.cc:1397] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:19:24.083155 26036 tablet_bootstrap.cc:492] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: No bootstrap required, opened a new log
I20260923 17:19:24.083155 26037 tablet_bootstrap.cc:492] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: No bootstrap required, opened a new log
I20260923 17:19:24.083328 26036 ts_tablet_manager.cc:1397] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:19:24.083328 26037 ts_tablet_manager.cc:1397] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:19:24.085820 26035 raft_consensus.cc:348] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.086144 26035 raft_consensus.cc:374] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.086202 26035 raft_consensus.cc:729] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66cdff20193405f8363abd92e5b2e98, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.086714 26035 consensus_queue.cc:260] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.087399 26012 heartbeater.cc:503] Master 127.23.214.190:43953 was elected leader, sending a full tablet report...
I20260923 17:19:24.087345 26037 raft_consensus.cc:348] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.087606 26037 raft_consensus.cc:374] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.087664 26037 raft_consensus.cc:729] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5e922a8dca47dcb40e355a2cf54428, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.087821 26035 ts_tablet_manager.cc:1428] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:19:24.088105 26037 consensus_queue.cc:260] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.088768 26036 raft_consensus.cc:348] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.089036 26036 raft_consensus.cc:374] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.089092 26036 raft_consensus.cc:729] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f421ed4943264178960a1c07ed0f14b1, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.089152 26037 ts_tablet_manager.cc:1428] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:19:24.089602 25872 heartbeater.cc:503] Master 127.23.214.190:43953 was elected leader, sending a full tablet report...
I20260923 17:19:24.089588 26036 consensus_queue.cc:260] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.090695 26036 ts_tablet_manager.cc:1428] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20260923 17:19:24.090839 25732 heartbeater.cc:503] Master 127.23.214.190:43953 was elected leader, sending a full tablet report...
I20260923 17:19:24.138022 26041 raft_consensus.cc:482] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:24.138262 26041 raft_consensus.cc:504] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.139406 26041 leader_election.cc:288] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f421ed4943264178960a1c07ed0f14b1 (127.23.214.129:44363), 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130:40199)
I20260923 17:19:24.144109 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.139673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:44363 (local address 127.23.214.131:38165)
0923 17:19:24.139881 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.139898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.140006 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.140317 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.140320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.140343 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.140489 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.140495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.141468 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.141470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.142927 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.142937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.143599 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.143603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.143759 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.143793 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.143852 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.143912 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":66}
I20260923 17:19:24.144125 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.139826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:38165 (local address 127.23.214.129:44363)
0923 17:19:24.139989 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.139994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.140012 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:19:24.140120 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.140123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.140205 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.140285 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.140613 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.141350 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.143058 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.143478 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.143630 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.143684 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.143896 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.143992 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:19:24.144147 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.139985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40199 (local address 127.23.214.131:53023)
0923 17:19:24.140385 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.140398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.140494 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.140935 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.140938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.140958 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.141110 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.141116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.142007 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.142012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.142938 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.142948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.143640 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.143644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.143822 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.143870 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.143950 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.144009 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":97,"thread_start_us":119,"threads_started":1}
I20260923 17:19:24.144178 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.140134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:53023 (local address 127.23.214.130:40199)
0923 17:19:24.140289 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.140294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.140509 (+ 215us) server_negotiation.cc:407] Connection header received
0923 17:19:24.140605 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.140608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.140656 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.140762 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.141231 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.141831 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.143101 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.143490 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.143625 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.143683 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.143988 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.144058 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:19:24.144843 25826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda5bc0635442789e34ca2a15153e1f" candidate_uuid: "f66cdff20193405f8363abd92e5b2e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5e922a8dca47dcb40e355a2cf54428" is_pre_election: true
I20260923 17:19:24.144865 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda5bc0635442789e34ca2a15153e1f" candidate_uuid: "f66cdff20193405f8363abd92e5b2e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f421ed4943264178960a1c07ed0f14b1" is_pre_election: true
I20260923 17:19:24.145154 25686 raft_consensus.cc:2454] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66cdff20193405f8363abd92e5b2e98 in term 0.
I20260923 17:19:24.145159 25826 raft_consensus.cc:2454] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66cdff20193405f8363abd92e5b2e98 in term 0.
I20260923 17:19:24.145670 25900 leader_election.cc:302] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f421ed4943264178960a1c07ed0f14b1, f66cdff20193405f8363abd92e5b2e98; no voters:
I20260923 17:19:24.145989 26041 raft_consensus.cc:2781] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:24.146088 26041 raft_consensus.cc:482] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:24.146167 26041 raft_consensus.cc:3037] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.147250 26041 raft_consensus.cc:504] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.147607 26041 leader_election.cc:288] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [CANDIDATE]: Term 1 election: Requested vote from peers f421ed4943264178960a1c07ed0f14b1 (127.23.214.129:44363), 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130:40199)
I20260923 17:19:24.148113 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda5bc0635442789e34ca2a15153e1f" candidate_uuid: "f66cdff20193405f8363abd92e5b2e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f421ed4943264178960a1c07ed0f14b1"
I20260923 17:19:24.148138 25826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda5bc0635442789e34ca2a15153e1f" candidate_uuid: "f66cdff20193405f8363abd92e5b2e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5e922a8dca47dcb40e355a2cf54428"
I20260923 17:19:24.148272 25826 raft_consensus.cc:3037] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.148240 25686 raft_consensus.cc:3037] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.149353 25826 raft_consensus.cc:2454] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66cdff20193405f8363abd92e5b2e98 in term 1.
I20260923 17:19:24.149400 25686 raft_consensus.cc:2454] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66cdff20193405f8363abd92e5b2e98 in term 1.
I20260923 17:19:24.149705 25900 leader_election.cc:302] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f421ed4943264178960a1c07ed0f14b1, f66cdff20193405f8363abd92e5b2e98; no voters:
I20260923 17:19:24.149964 26041 raft_consensus.cc:2781] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:24.150208 26041 raft_consensus.cc:686] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 1 LEADER]: Becoming Leader. State: Replica: f66cdff20193405f8363abd92e5b2e98, State: Running, Role: LEADER
I20260923 17:19:24.150610 26041 consensus_queue.cc:237] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } }
I20260923 17:19:24.154296 25529 catalog_manager.cc:5495] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to f66cdff20193405f8363abd92e5b2e98 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "f66cdff20193405f8363abd92e5b2e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } health_report { overall_health: UNKNOWN } } }
I20260923 17:19:24.194355 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.190103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.23.214.131:35621)
0923 17:19:24.190274 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.190278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.190344 (+ 66us) server_negotiation.cc:407] Connection header received
0923 17:19:24.190567 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.190570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.190623 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.190725 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:24.191147 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.191727 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.192696 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.192937 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.193056 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.193406 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.193898 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.194031 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.194122 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.194185 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:19:24.200404 25686 raft_consensus.cc:1261] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Refusing update from remote peer f66cdff20193405f8363abd92e5b2e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:24.200404 25826 raft_consensus.cc:1261] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Refusing update from remote peer f66cdff20193405f8363abd92e5b2e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:24.201195 26045 consensus_queue.cc:1035] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:24.201429 26041 consensus_queue.cc:1035] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:24.208413 26048 mvcc.cc:205] Tried to move back new op lower bound from 7332593517357297664 to 7332593517163839488. Current Snapshot: MvccSnapshot[applied={T|T < 7332593517357297664}]
I20260923 17:19:24.224193 25525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:24.224400 25525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:24.227284 25525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5: Sending DeleteTablet for 3 replicas of tablet 4cda5bc0635442789e34ca2a15153e1f
I20260923 17:19:24.228343 25806 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cda5bc0635442789e34ca2a15153e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:48826
I20260923 17:19:24.228502 25946 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cda5bc0635442789e34ca2a15153e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:57142
I20260923 17:19:24.229228 25666 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cda5bc0635442789e34ca2a15153e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:36806
I20260923 17:19:24.229226 25525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:24.229738 25525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:24.230147 26059 tablet_replica.cc:330] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: stopping tablet replica
I20260923 17:19:24.230538 26059 raft_consensus.cc:2229] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:24.231016 26059 raft_consensus.cc:2258] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:24.231527 26058 tablet_replica.cc:330] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: stopping tablet replica
I20260923 17:19:24.231842 26058 raft_consensus.cc:2229] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:24.234493 26059 ts_tablet_manager.cc:1905] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:24.235067 25666 tablet_service.cc:1467] Processing CreateTablet for tablet beee9adb9b5a4a9ca86d4552e8f53b7c (DEFAULT_TABLE table=test-table [id=771dac76c8014bddb43ff9dc73a76e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:24.235247 25946 tablet_service.cc:1467] Processing CreateTablet for tablet beee9adb9b5a4a9ca86d4552e8f53b7c (DEFAULT_TABLE table=test-table [id=771dac76c8014bddb43ff9dc73a76e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:24.235414 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beee9adb9b5a4a9ca86d4552e8f53b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.235509 25946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beee9adb9b5a4a9ca86d4552e8f53b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.237407 26036 tablet_bootstrap.cc:492] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Bootstrap starting.
I20260923 17:19:24.237622 26058 raft_consensus.cc:2258] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:24.238001 25806 tablet_service.cc:1467] Processing CreateTablet for tablet beee9adb9b5a4a9ca86d4552e8f53b7c (DEFAULT_TABLE table=test-table [id=771dac76c8014bddb43ff9dc73a76e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:24.238245 25806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beee9adb9b5a4a9ca86d4552e8f53b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.238469 26036 tablet_bootstrap.cc:654] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.239459 26058 ts_tablet_manager.cc:1905] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:24.240723 26037 tablet_bootstrap.cc:492] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Bootstrap starting.
I20260923 17:19:24.241266 26035 tablet_bootstrap.cc:492] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: Bootstrap starting.
I20260923 17:19:24.241683 26037 tablet_bootstrap.cc:654] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.242236 26059 ts_tablet_manager.cc:1918] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:24.242281 26035 tablet_bootstrap.cc:654] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.242327 26059 log.cc:1196] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/4cda5bc0635442789e34ca2a15153e1f
I20260923 17:19:24.242648 26059 ts_tablet_manager.cc:1939] T 4cda5bc0635442789e34ca2a15153e1f P f66cdff20193405f8363abd92e5b2e98: Deleting consensus metadata
I20260923 17:19:24.242820 26036 tablet_bootstrap.cc:492] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: No bootstrap required, opened a new log
I20260923 17:19:24.242906 26036 ts_tablet_manager.cc:1397] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 17:19:24.243345 26036 raft_consensus.cc:348] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.243463 26036 raft_consensus.cc:374] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.243494 26036 raft_consensus.cc:729] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f421ed4943264178960a1c07ed0f14b1, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.243788 25516 catalog_manager.cc:4841] TS f66cdff20193405f8363abd92e5b2e98 (127.23.214.131:35621): tablet 4cda5bc0635442789e34ca2a15153e1f (table test-table [id=acb2b59e8f0f4d088bdd10c400d5fb62]) successfully deleted
I20260923 17:19:24.243613 26036 consensus_queue.cc:260] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.245098 26036 ts_tablet_manager.cc:1428] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 17:19:24.245694 26058 ts_tablet_manager.cc:1918] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:24.245791 26058 log.cc:1196] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/4cda5bc0635442789e34ca2a15153e1f
I20260923 17:19:24.245930 26037 tablet_bootstrap.cc:492] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: No bootstrap required, opened a new log
I20260923 17:19:24.246008 26037 ts_tablet_manager.cc:1397] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260923 17:19:24.246086 26060 tablet_replica.cc:330] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: stopping tablet replica
I20260923 17:19:24.246388 26060 raft_consensus.cc:2229] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:24.246423 26037 raft_consensus.cc:348] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.246533 26037 raft_consensus.cc:374] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.246568 26037 raft_consensus.cc:729] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5e922a8dca47dcb40e355a2cf54428, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.246701 26037 consensus_queue.cc:260] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.246739 26060 raft_consensus.cc:2258] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:24.246979 26037 ts_tablet_manager.cc:1428] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:19:24.247272 26058 ts_tablet_manager.cc:1939] T 4cda5bc0635442789e34ca2a15153e1f P 0b5e922a8dca47dcb40e355a2cf54428: Deleting consensus metadata
I20260923 17:19:24.248558 25516 catalog_manager.cc:4841] TS 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130:40199): tablet 4cda5bc0635442789e34ca2a15153e1f (table test-table [id=acb2b59e8f0f4d088bdd10c400d5fb62]) successfully deleted
I20260923 17:19:24.249708 26060 ts_tablet_manager.cc:1905] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:24.251838 26060 ts_tablet_manager.cc:1918] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:24.251950 26060 log.cc:1196] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/4cda5bc0635442789e34ca2a15153e1f
I20260923 17:19:24.252777 26060 ts_tablet_manager.cc:1939] T 4cda5bc0635442789e34ca2a15153e1f P f421ed4943264178960a1c07ed0f14b1: Deleting consensus metadata
I20260923 17:19:24.253823 25516 catalog_manager.cc:4841] TS f421ed4943264178960a1c07ed0f14b1 (127.23.214.129:44363): tablet 4cda5bc0635442789e34ca2a15153e1f (table test-table [id=acb2b59e8f0f4d088bdd10c400d5fb62]) successfully deleted
I20260923 17:19:24.255211 26042 raft_consensus.cc:482] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:24.255334 26042 raft_consensus.cc:504] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.256554 26042 leader_election.cc:288] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f421ed4943264178960a1c07ed0f14b1 (127.23.214.129:44363), f66cdff20193405f8363abd92e5b2e98 (127.23.214.131:35621)
I20260923 17:19:24.258749 26035 tablet_bootstrap.cc:492] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: No bootstrap required, opened a new log
I20260923 17:19:24.258858 26035 ts_tablet_manager.cc:1397] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20260923 17:19:24.259387 26035 raft_consensus.cc:348] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.259500 26035 raft_consensus.cc:374] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.259527 26035 raft_consensus.cc:729] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66cdff20193405f8363abd92e5b2e98, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.259646 26035 consensus_queue.cc:260] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.259902 26035 ts_tablet_manager.cc:1428] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:24.261265 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.256756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:44363 (local address 127.23.214.130:59623)
0923 17:19:24.256966 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.256982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.257062 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.257437 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.257440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.257457 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.257579 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.257584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.258634 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.258638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.259915 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.259925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.260829 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.260833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.260936 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.260957 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.261031 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.261121 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260923 17:19:24.261921 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.257121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:57393 (local address 127.23.214.131:35621)
0923 17:19:24.257234 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.257239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.257483 (+ 244us) server_negotiation.cc:407] Connection header received
0923 17:19:24.257671 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.257677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.257729 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.257823 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.258116 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.258773 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.260245 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.260688 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.260823 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.260882 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.261675 (+ 793us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.261753 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":44}
I20260923 17:19:24.261925 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.256807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:59623 (local address 127.23.214.129:44363)
0923 17:19:24.257225 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.257231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.257244 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:19:24.257287 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.257291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.257333 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.257426 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.257701 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.258505 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.260073 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.260687 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.260820 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.260876 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.261723 (+ 847us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.261798 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260923 17:19:24.262296 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.257012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:35621 (local address 127.23.214.130:57393)
0923 17:19:24.257397 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.257411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.257547 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.257832 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.257835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.257851 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.258009 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.258014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.258916 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.258919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.260111 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.260122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.261412 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.261416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.261533 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.261567 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.262087 (+ 520us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.262166 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":87,"threads_started":1}
I20260923 17:19:24.262276 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beee9adb9b5a4a9ca86d4552e8f53b7c" candidate_uuid: "0b5e922a8dca47dcb40e355a2cf54428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f421ed4943264178960a1c07ed0f14b1" is_pre_election: true
I20260923 17:19:24.262437 25686 raft_consensus.cc:2454] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b5e922a8dca47dcb40e355a2cf54428 in term 0.
I20260923 17:19:24.262817 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beee9adb9b5a4a9ca86d4552e8f53b7c" candidate_uuid: "0b5e922a8dca47dcb40e355a2cf54428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66cdff20193405f8363abd92e5b2e98" is_pre_election: true
I20260923 17:19:24.262835 25760 leader_election.cc:302] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5e922a8dca47dcb40e355a2cf54428, f421ed4943264178960a1c07ed0f14b1; no voters:
I20260923 17:19:24.263046 25956 raft_consensus.cc:2454] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b5e922a8dca47dcb40e355a2cf54428 in term 0.
W20260923 17:19:24.263100 25873 tablet.cc:2369] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:24.263110 26042 raft_consensus.cc:2781] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:24.263674 26042 raft_consensus.cc:482] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:24.263720 26042 raft_consensus.cc:3037] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.264665 26042 raft_consensus.cc:504] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.265057 26042 leader_election.cc:288] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [CANDIDATE]: Term 1 election: Requested vote from peers f421ed4943264178960a1c07ed0f14b1 (127.23.214.129:44363), f66cdff20193405f8363abd92e5b2e98 (127.23.214.131:35621)
I20260923 17:19:24.265467 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beee9adb9b5a4a9ca86d4552e8f53b7c" candidate_uuid: "0b5e922a8dca47dcb40e355a2cf54428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66cdff20193405f8363abd92e5b2e98"
I20260923 17:19:24.265555 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beee9adb9b5a4a9ca86d4552e8f53b7c" candidate_uuid: "0b5e922a8dca47dcb40e355a2cf54428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f421ed4943264178960a1c07ed0f14b1"
I20260923 17:19:24.265600 25956 raft_consensus.cc:3037] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.265684 25686 raft_consensus.cc:3037] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.266865 25956 raft_consensus.cc:2454] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5e922a8dca47dcb40e355a2cf54428 in term 1.
I20260923 17:19:24.266868 25686 raft_consensus.cc:2454] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5e922a8dca47dcb40e355a2cf54428 in term 1.
I20260923 17:19:24.267125 25760 leader_election.cc:302] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5e922a8dca47dcb40e355a2cf54428, f66cdff20193405f8363abd92e5b2e98; no voters:
I20260923 17:19:24.267306 26042 raft_consensus.cc:2781] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:24.267536 26042 raft_consensus.cc:686] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 LEADER]: Becoming Leader. State: Replica: 0b5e922a8dca47dcb40e355a2cf54428, State: Running, Role: LEADER
I20260923 17:19:24.267798 26042 consensus_queue.cc:237] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
W20260923 17:19:24.269483 25733 tablet.cc:2369] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:24.270152 25525 catalog_manager.cc:5495] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130). New cstate: current_term: 1 leader_uuid: "0b5e922a8dca47dcb40e355a2cf54428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:19:24 dist-test-slave-9q7f krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183963, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:24.304988 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.295700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43953 (local address 127.0.0.1:41796)
0923 17:19:24.296003 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.296020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.296135 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.296420 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.296423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.296830 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:24.296922 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.296928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.298077 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.298079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.298678 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.298686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.299241 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.299244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.299304 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.299951 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:24.299963 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:24.301605 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:24.303465 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:24.303473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:24.303478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:24.303729 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:24.304005 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:24.304007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:24.304009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:24.304082 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:24.304488 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:24.304493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:24.304617 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.304725 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.304851 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
I20260923 17:19:24.305106 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.295790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.23.214.190:43953)
0923 17:19:24.295953 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.295959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.296106 (+ 147us) server_negotiation.cc:407] Connection header received
0923 17:19:24.296224 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.296228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.296287 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.296389 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:24.297048 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.297958 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.298785 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.299125 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.299222 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.301741 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:24.301745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:24.301748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:24.301773 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:24.303349 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.303822 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.303826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.303828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.303918 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.304163 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.304166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.304168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.304388 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:24.304485 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.304803 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.304927 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:19:24.309310 25525 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:41796:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:19:24.310158 25525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:24.314998 25806 tablet_service.cc:1467] Processing CreateTablet for tablet b367028e7b814a4a9ba9e7be4ae8f17c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e3053b246874c16951091ac74725e50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:24.314998 25666 tablet_service.cc:1467] Processing CreateTablet for tablet b367028e7b814a4a9ba9e7be4ae8f17c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e3053b246874c16951091ac74725e50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:24.315086 25946 tablet_service.cc:1467] Processing CreateTablet for tablet b367028e7b814a4a9ba9e7be4ae8f17c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e3053b246874c16951091ac74725e50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:24.315407 25806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b367028e7b814a4a9ba9e7be4ae8f17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.315407 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b367028e7b814a4a9ba9e7be4ae8f17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.315475 25946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b367028e7b814a4a9ba9e7be4ae8f17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:24.318395 26035 tablet_bootstrap.cc:492] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: Bootstrap starting.
I20260923 17:19:24.318473 26036 tablet_bootstrap.cc:492] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: Bootstrap starting.
I20260923 17:19:24.319262 26035 tablet_bootstrap.cc:654] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.319423 26036 tablet_bootstrap.cc:654] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.320727 26035 tablet_bootstrap.cc:492] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: No bootstrap required, opened a new log
I20260923 17:19:24.320820 26035 ts_tablet_manager.cc:1397] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:24.320996 26036 tablet_bootstrap.cc:492] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: No bootstrap required, opened a new log
I20260923 17:19:24.321067 26036 ts_tablet_manager.cc:1397] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260923 17:19:24.321151 26035 raft_consensus.cc:348] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.321237 26035 raft_consensus.cc:374] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.321257 26035 raft_consensus.cc:729] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66cdff20193405f8363abd92e5b2e98, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.321406 26035 consensus_queue.cc:260] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.321480 26036 raft_consensus.cc:348] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.321580 26036 raft_consensus.cc:374] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.321609 26036 raft_consensus.cc:729] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f421ed4943264178960a1c07ed0f14b1, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.321695 26035 ts_tablet_manager.cc:1428] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:24.321632 26054 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.321748 26036 consensus_queue.cc:260] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.321784 26054 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:24.322062 26043 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.322202 26043 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:24.322678 26037 tablet_bootstrap.cc:492] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: Bootstrap starting.
I20260923 17:19:24.322829 26036 ts_tablet_manager.cc:1428] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 17:19:24.323688 26037 tablet_bootstrap.cc:654] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:24.325191 26037 tablet_bootstrap.cc:492] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: No bootstrap required, opened a new log
I20260923 17:19:24.325304 26037 ts_tablet_manager.cc:1397] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 17:19:24.325726 26037 raft_consensus.cc:348] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.325914 26037 raft_consensus.cc:374] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:24.325959 26037 raft_consensus.cc:729] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5e922a8dca47dcb40e355a2cf54428, State: Initialized, Role: FOLLOWER
I20260923 17:19:24.326165 26037 consensus_queue.cc:260] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.326465 26037 ts_tablet_manager.cc:1428] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 17:19:24.326673 26042 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.326802 26042 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 17:19:24.503322 26013 tablet.cc:2369] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:24.536624 26043 raft_consensus.cc:482] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:24.536759 26043 raft_consensus.cc:504] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.537947 26043 leader_election.cc:288] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130:40199), f66cdff20193405f8363abd92e5b2e98 (127.23.214.131:35621)
I20260923 17:19:24.543093 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.538354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:50053 (local address 127.23.214.130:40199)
0923 17:19:24.538539 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.538544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.538562 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:19:24.538731 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.538734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.538784 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.538861 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.539155 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.539972 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.541742 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.542408 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.542541 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.542592 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.542881 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.542953 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:19:24.543093 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.538731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:58965 (local address 127.23.214.131:35621)
0923 17:19:24.538867 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.538872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.539018 (+ 146us) server_negotiation.cc:407] Connection header received
0923 17:19:24.539143 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.539146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.539192 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.539268 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:24.539463 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.540217 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.541743 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.542408 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.542541 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.542592 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.542881 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.542954 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 17:19:24.543148 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.538592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:35621 (local address 127.23.214.129:58965)
0923 17:19:24.538922 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.538935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.539044 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.539285 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.539287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.539300 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.539363 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.539367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.540333 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.540337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.541628 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.541638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.542561 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.542564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.542724 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.542772 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.542853 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.542959 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":73,"thread_start_us":90,"threads_started":1}
I20260923 17:19:24.543156 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.538204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40199 (local address 127.23.214.129:50053)
0923 17:19:24.538467 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.538480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.538616 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.538896 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.538900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.538921 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:24.539050 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.539056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.540090 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.540094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.541560 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.541573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.542545 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.542553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.542753 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.542773 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.542853 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.542959 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":94,"thread_start_us":76,"threads_started":1}
I20260923 17:19:24.543850 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b367028e7b814a4a9ba9e7be4ae8f17c" candidate_uuid: "f421ed4943264178960a1c07ed0f14b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66cdff20193405f8363abd92e5b2e98" is_pre_election: true
I20260923 17:19:24.543850 25826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b367028e7b814a4a9ba9e7be4ae8f17c" candidate_uuid: "f421ed4943264178960a1c07ed0f14b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5e922a8dca47dcb40e355a2cf54428" is_pre_election: true
I20260923 17:19:24.544010 25826 raft_consensus.cc:2454] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f421ed4943264178960a1c07ed0f14b1 in term 0.
I20260923 17:19:24.544010 25956 raft_consensus.cc:2454] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f421ed4943264178960a1c07ed0f14b1 in term 0.
I20260923 17:19:24.544421 25620 leader_election.cc:302] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f421ed4943264178960a1c07ed0f14b1, f66cdff20193405f8363abd92e5b2e98; no voters:
I20260923 17:19:24.544701 26043 raft_consensus.cc:2781] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:24.544754 26043 raft_consensus.cc:482] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:24.544781 26043 raft_consensus.cc:3037] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.545818 26043 raft_consensus.cc:504] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.546229 26043 leader_election.cc:288] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [CANDIDATE]: Term 1 election: Requested vote from peers 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130:40199), f66cdff20193405f8363abd92e5b2e98 (127.23.214.131:35621)
I20260923 17:19:24.546638 25826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b367028e7b814a4a9ba9e7be4ae8f17c" candidate_uuid: "f421ed4943264178960a1c07ed0f14b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5e922a8dca47dcb40e355a2cf54428"
I20260923 17:19:24.546761 25826 raft_consensus.cc:3037] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.546715 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b367028e7b814a4a9ba9e7be4ae8f17c" candidate_uuid: "f421ed4943264178960a1c07ed0f14b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66cdff20193405f8363abd92e5b2e98"
I20260923 17:19:24.546908 25956 raft_consensus.cc:3037] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:24.547760 25826 raft_consensus.cc:2454] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f421ed4943264178960a1c07ed0f14b1 in term 1.
I20260923 17:19:24.547842 25956 raft_consensus.cc:2454] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f421ed4943264178960a1c07ed0f14b1 in term 1.
I20260923 17:19:24.548116 25620 leader_election.cc:302] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5e922a8dca47dcb40e355a2cf54428, f421ed4943264178960a1c07ed0f14b1; no voters:
I20260923 17:19:24.548274 26043 raft_consensus.cc:2781] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:24.548475 26043 raft_consensus.cc:686] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [term 1 LEADER]: Becoming Leader. State: Replica: f421ed4943264178960a1c07ed0f14b1, State: Running, Role: LEADER
I20260923 17:19:24.548753 26043 consensus_queue.cc:237] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } }
I20260923 17:19:24.550076 26078 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: TxnStatusTablet state changed. Reason: New leader f421ed4943264178960a1c07ed0f14b1. Latest consensus state: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.550194 26078 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:24.550693 26080 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:19:24.550848 26080 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:19:24.551489 25525 catalog_manager.cc:5495] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f421ed4943264178960a1c07ed0f14b1 (127.23.214.129). New cstate: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } health_report { overall_health: UNKNOWN } } }
I20260923 17:19:24.563586 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.559552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.23.214.129:44363)
0923 17:19:24.559690 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.559694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.559797 (+ 103us) server_negotiation.cc:407] Connection header received
0923 17:19:24.559987 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.559989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.560042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.560121 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:24.560403 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.560968 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.562196 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.562413 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.562489 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.562804 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.563203 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.563292 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.563417 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.563467 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:19:24.563629 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.559479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:44363 (local address 127.0.0.1:36822)
0923 17:19:24.559725 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.559741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.559868 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.560161 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.560164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.560173 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:24.560302 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.560310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.561072 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.561077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.562082 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.562093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.562520 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.562524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.562618 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.562674 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.563296 (+ 622us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.563311 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.563385 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.563471 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 17:19:24.566975 25826 raft_consensus.cc:1261] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Refusing update from remote peer f421ed4943264178960a1c07ed0f14b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:24.567221 25956 raft_consensus.cc:1261] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Refusing update from remote peer f421ed4943264178960a1c07ed0f14b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:24.567495 26043 consensus_queue.cc:1035] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:24.567703 26078 consensus_queue.cc:1035] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:24.569363 26042 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: TxnStatusTablet state changed. Reason: New leader f421ed4943264178960a1c07ed0f14b1. Latest consensus state: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.569491 26042 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:24.569629 26054 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: TxnStatusTablet state changed. Reason: New leader f421ed4943264178960a1c07ed0f14b1. Latest consensus state: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.569739 26054 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:24.571118 26043 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.571236 26043 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:24.572669 26049 mvcc.cc:205] Tried to move back new op lower bound from 7332593518860115968 to 7332593518794633216. Current Snapshot: MvccSnapshot[applied={T|T < 7332593518860115968 or (T in {7332593518860115968})}]
I20260923 17:19:24.573786 26054 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.573932 26054 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P f66cdff20193405f8363abd92e5b2e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:24.579416 26078 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.579572 26078 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P f421ed4943264178960a1c07ed0f14b1: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:24.580111 26050 mvcc.cc:205] Tried to move back new op lower bound from 7332593518860115968 to 7332593518794633216. Current Snapshot: MvccSnapshot[applied={T|T < 7332593518860115968}]
I20260923 17:19:24.580283 26042 tablet_replica.cc:439] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f421ed4943264178960a1c07ed0f14b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 } } peers { permanent_uuid: "0b5e922a8dca47dcb40e355a2cf54428" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40199 } } peers { permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 } } }
I20260923 17:19:24.580387 26042 tablet_replica.cc:442] T b367028e7b814a4a9ba9e7be4ae8f17c P 0b5e922a8dca47dcb40e355a2cf54428: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:24.582114 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.576338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.23.214.130:40199)
0923 17:19:24.576647 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.576652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.576664 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:19:24.576744 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.576748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.576824 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.576933 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:24.577202 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.578041 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.579044 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.579338 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.579449 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.579953 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.580543 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.580667 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.581933 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.581992 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260923 17:19:24.594298 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.589458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:48335 (local address 127.23.214.129:44363)
0923 17:19:24.589595 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.589600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.589801 (+ 201us) server_negotiation.cc:407] Connection header received
0923 17:19:24.589971 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.589973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.590046 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.590131 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:24.590439 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.591018 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.592867 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.593373 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.593477 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.593818 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.593955 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.594016 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.594140 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.594186 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:19:24.594372 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.589366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:44363 (local address 127.23.214.130:48335)
0923 17:19:24.589683 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.589699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.589819 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.590160 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.590163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.590173 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:24.590300 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.590307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.591118 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.591122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.592725 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.592736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.593522 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.593526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.593622 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.593679 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.594045 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.594052 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.594123 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.594204 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":118,"threads_started":1}
I20260923 17:19:24.601316 25956 raft_consensus.cc:1261] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Refusing update from remote peer 0b5e922a8dca47dcb40e355a2cf54428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:24.601420 25686 raft_consensus.cc:1261] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Refusing update from remote peer 0b5e922a8dca47dcb40e355a2cf54428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:24.601799 26065 consensus_queue.cc:1035] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66cdff20193405f8363abd92e5b2e98" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 35621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:24.602219 26065 consensus_queue.cc:1035] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f421ed4943264178960a1c07ed0f14b1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 44363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:24.623639 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.619472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:50353 (local address 127.23.214.130:40199)
0923 17:19:24.619727 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.619732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.619840 (+ 108us) server_negotiation.cc:407] Connection header received
0923 17:19:24.620001 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.620004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.620048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.620154 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:24.620451 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.621196 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.622510 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.622718 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.622800 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.623140 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.623235 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.623305 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.623452 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.623507 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260923 17:19:24.623674 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.619437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40199 (local address 127.23.214.129:50353)
0923 17:19:24.619743 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:24.619759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:24.619894 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:24.620166 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:24.620169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:24.620177 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:24.620306 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.620313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.621312 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.621316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.622386 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:24.622398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.622850 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.622854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:24.622953 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.623008 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:24.623326 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:24.623332 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:19:24.623423 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 17:19:24.623502 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
I20260923 17:19:24.650362 25529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:24.650549 25529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:24.653071 25529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb7aaafa704c4491bf96737634d66eb5: Sending DeleteTablet for 3 replicas of tablet beee9adb9b5a4a9ca86d4552e8f53b7c
I20260923 17:19:24.654078 25946 tablet_service.cc:1514] Processing DeleteTablet for tablet beee9adb9b5a4a9ca86d4552e8f53b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:57142
I20260923 17:19:24.654263 26059 tablet_replica.cc:330] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: stopping tablet replica
I20260923 17:19:24.654376 26059 raft_consensus.cc:2229] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:24.654487 26059 raft_consensus.cc:2258] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:24.654650 25666 tablet_service.cc:1514] Processing DeleteTablet for tablet beee9adb9b5a4a9ca86d4552e8f53b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:36806
I20260923 17:19:24.654820 26060 tablet_replica.cc:330] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: stopping tablet replica
I20260923 17:19:24.654922 26060 raft_consensus.cc:2229] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:24.655019 26060 raft_consensus.cc:2258] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:24.655834 26060 ts_tablet_manager.cc:1905] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:24.655844 26059 ts_tablet_manager.cc:1905] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:24.658705 26059 ts_tablet_manager.cc:1918] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:24.658699 26060 ts_tablet_manager.cc:1918] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:24.658790 26059 log.cc:1196] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/beee9adb9b5a4a9ca86d4552e8f53b7c
I20260923 17:19:24.658778 26060 log.cc:1196] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/beee9adb9b5a4a9ca86d4552e8f53b7c
I20260923 17:19:24.659058 26059 ts_tablet_manager.cc:1939] T beee9adb9b5a4a9ca86d4552e8f53b7c P f66cdff20193405f8363abd92e5b2e98: Deleting consensus metadata
I20260923 17:19:24.659061 26060 ts_tablet_manager.cc:1939] T beee9adb9b5a4a9ca86d4552e8f53b7c P f421ed4943264178960a1c07ed0f14b1: Deleting consensus metadata
I20260923 17:19:24.660004 25516 catalog_manager.cc:4841] TS f421ed4943264178960a1c07ed0f14b1 (127.23.214.129:44363): tablet beee9adb9b5a4a9ca86d4552e8f53b7c (table test-table [id=771dac76c8014bddb43ff9dc73a76e14]) successfully deleted
I20260923 17:19:24.660347 25516 catalog_manager.cc:4841] TS f66cdff20193405f8363abd92e5b2e98 (127.23.214.131:35621): tablet beee9adb9b5a4a9ca86d4552e8f53b7c (table test-table [id=771dac76c8014bddb43ff9dc73a76e14]) successfully deleted
I20260923 17:19:24.660730 25806 tablet_service.cc:1514] Processing DeleteTablet for tablet beee9adb9b5a4a9ca86d4552e8f53b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:48826
I20260923 17:19:24.660930 26058 tablet_replica.cc:330] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: stopping tablet replica
I20260923 17:19:24.661041 26058 raft_consensus.cc:2229] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:24.661365 26058 raft_consensus.cc:2258] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:24.662213 26058 ts_tablet_manager.cc:1905] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:24.665083 26058 ts_tablet_manager.cc:1918] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:24.665158 26058 log.cc:1196] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/beee9adb9b5a4a9ca86d4552e8f53b7c
I20260923 17:19:24.665428 26058 ts_tablet_manager.cc:1939] T beee9adb9b5a4a9ca86d4552e8f53b7c P 0b5e922a8dca47dcb40e355a2cf54428: Deleting consensus metadata
I20260923 17:19:24.666570 25516 catalog_manager.cc:4841] TS 0b5e922a8dca47dcb40e355a2cf54428 (127.23.214.130:40199): tablet beee9adb9b5a4a9ca86d4552e8f53b7c (table test-table [id=771dac76c8014bddb43ff9dc73a76e14]) successfully deleted
Sep 23 17:19:24 dist-test-slave-9q7f krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183964, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:24.674860 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.661969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.23.214.129:44363)
0923 17:19:24.662082 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.662087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.662664 (+ 577us) server_negotiation.cc:407] Connection header received
0923 17:19:24.662862 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.662866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.662911 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.662961 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:24.663968 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.664746 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.666056 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.666389 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.666470 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.670467 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:24.670477 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:24.670495 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:24.670528 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:24.672674 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.673214 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.673220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.673222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.673316 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.673623 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.673627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.673628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.673954 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:24.674142 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.674544 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.674684 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260923 17:19:24.675813 25715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36836
I20260923 17:19:24.687484 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:24.678107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.23.214.190:43953)
0923 17:19:24.678235 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:24.678240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:24.678481 (+ 241us) server_negotiation.cc:407] Connection header received
0923 17:19:24.678583 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:24.678586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:24.678638 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:24.678722 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:24.679526 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.680385 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.681552 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:24.681948 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:24.682051 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:24.683776 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:24.683780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:24.683791 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:24.683825 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:24.685594 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.686138 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.686142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.686144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.686237 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:24.686545 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:24.686549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:24.686550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:24.686792 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:24.686921 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:24.687177 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 17:19:24.687303 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 17:19:24.688251 25529 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41810
I20260923 17:19:24.690042 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 25601
I20260923 17:19:24.697942 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 25741
I20260923 17:19:24.703913 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 25881
I20260923 17:19:24.710925 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 25497
2026-09-23T17:19:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3342 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:24 dist-test-slave-9q7f krb5kdc[26102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:24 dist-test-slave-9q7f krb5kdc[26102](info): set up 2 sockets
Sep 23 17:19:24 dist-test-slave-9q7f krb5kdc[26102](info): commencing operation
krb5kdc: starting...
W20260923 17:19:25.760142 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:25 dist-test-slave-9q7f krb5kdc[26102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:25.894491 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35007
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:26.009577 26115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:26.009977 26115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:26.010075 26115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:26.014294 26115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:26.014377 26115 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:19:26.014396 26115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:26.014415 26115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:26.014432 26115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:26.014482 26115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:26.020150 26115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:35007
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:26.021301 26115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:26.022426 26115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:26.031250 26123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:26.031267 26121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:26.031400 26115 server_base.cc:1034] running on GCE node
W20260923 17:19:26.031267 26120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:26.032099 26115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:26.033239 26115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:26.034428 26115 hybrid_clock.cc:648] HybridClock initialized: now 1790183966034414 us; error 25 us; skew 500 ppm
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:26.037211 26115 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:26.038203 26115 webserver.cc:460] Webserver started at http://127.23.214.190:42815/ using document root <none> and password file <none>
I20260923 17:19:26.038769 26115 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:26.038877 26115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:26.039115 26115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:26.041257 26115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "849052ab5b8842b2993f682b85626d56"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "4266dbe3732ad3c79be8d909da45b458"
server_key_iv: "4e03abfd8d277cb344edd2895689f22f"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.041839 26115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "849052ab5b8842b2993f682b85626d56"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "4266dbe3732ad3c79be8d909da45b458"
server_key_iv: "4e03abfd8d277cb344edd2895689f22f"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.045653 26115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:26.048214 26130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:26.049602 26115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:26.049769 26115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "849052ab5b8842b2993f682b85626d56"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "4266dbe3732ad3c79be8d909da45b458"
server_key_iv: "4e03abfd8d277cb344edd2895689f22f"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.049943 26115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:26.072973 26115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:26.076238 26115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:26.076465 26115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:26.085155 26115 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:35007
I20260923 17:19:26.085175 26192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:35007 every 8 connection(s)
I20260923 17:19:26.086268 26115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:26.089157 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:26.091439 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 26115
I20260923 17:19:26.091562 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:26.091825 24410 external_mini_cluster.cc:1428] Setting key 684cf1c95900f9edb1c2f323f06f9e72
I20260923 17:19:26.095149 26193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Bootstrap starting.
I20260923 17:19:26.097325 26193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:26.098112 26193 log.cc:824] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:26.099942 26193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: No bootstrap required, opened a new log
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:26.103847 26193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849052ab5b8842b2993f682b85626d56" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35007 } }
I20260923 17:19:26.104193 26193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:26.104275 26193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849052ab5b8842b2993f682b85626d56, State: Initialized, Role: FOLLOWER
I20260923 17:19:26.104764 26193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849052ab5b8842b2993f682b85626d56" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35007 } }
I20260923 17:19:26.104943 26193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:26.105021 26193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:26.105125 26193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:26.106179 26193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849052ab5b8842b2993f682b85626d56" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35007 } }
I20260923 17:19:26.106534 26193 leader_election.cc:302] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 849052ab5b8842b2993f682b85626d56; no voters:
I20260923 17:19:26.106566 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.093257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39778 (local address 127.23.214.190:35007)
0923 17:19:26.093698 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.093714 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.093749 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:19:26.094442 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.094458 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.094801 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.095134 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.095909 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.096914 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.097818 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.098402 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.098575 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.101398 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.101403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.101416 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.101443 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.103228 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.103717 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.103722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.103726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.103842 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.104139 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.104141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.104142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.104754 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.104945 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.105319 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.105648 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260923 17:19:26.106907 26193 leader_election.cc:288] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:26.106980 26198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:26.107196 26198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [term 1 LEADER]: Becoming Leader. State: Replica: 849052ab5b8842b2993f682b85626d56, State: Running, Role: LEADER
I20260923 17:19:26.107492 26198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849052ab5b8842b2993f682b85626d56" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35007 } }
I20260923 17:19:26.109169 26200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "849052ab5b8842b2993f682b85626d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849052ab5b8842b2993f682b85626d56" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35007 } } }
I20260923 17:19:26.109378 26200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:26.109416 26199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 849052ab5b8842b2993f682b85626d56. Latest consensus state: current_term: 1 leader_uuid: "849052ab5b8842b2993f682b85626d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849052ab5b8842b2993f682b85626d56" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35007 } } }
I20260923 17:19:26.109524 26199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:26.109266 26193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:26.109797 26202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:26.112974 26202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:26.118808 26202 catalog_manager.cc:1347] Generated new cluster ID: f75a54d96ea54cbbbf8ea47d18574d41
I20260923 17:19:26.118901 26202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:26.137543 26202 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:26.138645 26202 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:26.148761 26202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Generated new TSK 0
I20260923 17:19:26.149708 26202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:26.190166 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35007
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:26.337090 26219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:26.337386 26219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:26.337437 26219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:26.341509 26219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:26.341580 26219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:26.341624 26219 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:19:26.341638 26219 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:26.341671 26219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:26.347107 26219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:26.348202 26219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:26.349224 26219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:26.361418 26224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:26.361387 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:26.361418 26225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:26.361755 26219 server_base.cc:1034] running on GCE node
I20260923 17:19:26.362234 26219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:26.362958 26219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:26.364158 26219 hybrid_clock.cc:648] HybridClock initialized: now 1790183966364138 us; error 33 us; skew 500 ppm
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:26.367265 26219 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:26.368275 26219 webserver.cc:460] Webserver started at http://127.23.214.129:38485/ using document root <none> and password file <none>
I20260923 17:19:26.368898 26219 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:26.368995 26219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:26.369242 26219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:26.371567 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "dc811c7cf7a44433b7e3e601d5ec5702"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "30b8e1ba075840e5972cbb021a934b5e"
server_key_iv: "b92d137bcc59445cf3ad24c780f7bf25"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.372156 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "dc811c7cf7a44433b7e3e601d5ec5702"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "30b8e1ba075840e5972cbb021a934b5e"
server_key_iv: "b92d137bcc59445cf3ad24c780f7bf25"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.376257 26219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:26.378937 26234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:26.380306 26219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:26.380460 26219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "dc811c7cf7a44433b7e3e601d5ec5702"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "30b8e1ba075840e5972cbb021a934b5e"
server_key_iv: "b92d137bcc59445cf3ad24c780f7bf25"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.380587 26219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:26.399308 26219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:26.402717 26219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:26.402944 26219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:26.404600 26219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:26.404677 26219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:26.404771 26219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:26.404798 26219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:26.416137 26219 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:37057
I20260923 17:19:26.416158 26349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:37057 every 8 connection(s)
I20260923 17:19:26.417141 26219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:26.418530 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 26219
I20260923 17:19:26.418664 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:26.418999 24410 external_mini_cluster.cc:1428] Setting key 1a92cb902d726acfbd06912830b96174
I20260923 17:19:26.420991 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.405333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:49587 (local address 127.23.214.190:35007)
0923 17:19:26.405475 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.405479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.407353 (+ 1874us) server_negotiation.cc:407] Connection header received
0923 17:19:26.408769 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.408773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.408834 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.408939 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.410514 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.411505 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.412793 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.413164 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.413306 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.416506 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.416510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.416514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.416555 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.418419 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.419131 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.419139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.419140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.419223 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.419659 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.419663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.419665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.419938 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.420051 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.420672 (+ 621us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.420805 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:19:26.421993 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.406536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35007 (local address 127.23.214.129:49587)
0923 17:19:26.407147 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.407229 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.408537 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.409119 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.409126 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.409641 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:26.410333 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.410354 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.411625 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.411627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.412671 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.412681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.413387 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.413390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.413556 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.414347 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:26.414375 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:26.416212 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:26.418546 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.418556 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.418574 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.418990 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.419323 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.419326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.419328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.419533 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.420056 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:26.420061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:26.420385 (+ 324us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.420622 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.420972 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":146,"threads_started":1}
I20260923 17:19:26.423532 26350 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35007
I20260923 17:19:26.423677 26350 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:26.424253 26350 heartbeater.cc:511] Master 127.23.214.190:35007 requested a full tablet report, sending...
I20260923 17:19:26.426231 26146 ts_manager.cc:194] Registered new tserver with Master: dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057)
I20260923 17:19:26.428190 26146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:49587
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:26.436507 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.425884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35007 (local address 127.23.214.129:39339)
0923 17:19:26.426309 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.426323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.426481 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.426752 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.426755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.426979 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:26.427079 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.427085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.430018 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.430022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.430839 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.430852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.431533 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.431536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.431643 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.432198 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:26.432210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:26.432673 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:26.434842 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.434847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.434849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.435069 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.435390 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.435393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.435395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.435491 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.435909 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:26.435913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:26.436017 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.436126 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.436311 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260923 17:19:26.436502 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.425984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:39339 (local address 127.23.214.190:35007)
0923 17:19:26.426111 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.426115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.426408 (+ 293us) server_negotiation.cc:407] Connection header received
0923 17:19:26.426596 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.426599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.426645 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.426737 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.427200 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.429254 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.430998 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.431375 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.431483 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.432836 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.432839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.432842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.432869 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.434699 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.435178 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.435181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.435183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.435278 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.435591 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.435594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.435596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.435802 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.435908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.436186 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.436311 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1371,"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:26.454365 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35007
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:26.572635 26359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:26.572947 26359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:26.573028 26359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:26.577289 26359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:26.577368 26359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:26.577410 26359 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:19:26.577425 26359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:26.577503 26359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:26.583155 26359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:26.584424 26359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:26.585616 26359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:26.596155 26365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:26.596187 26367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:26.596170 26364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:26.596498 26359 server_base.cc:1034] running on GCE node
I20260923 17:19:26.597045 26359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:26.597787 26359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:26.599031 26359 hybrid_clock.cc:648] HybridClock initialized: now 1790183966598992 us; error 51 us; skew 500 ppm
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:26.602114 26359 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:26.603173 26359 webserver.cc:460] Webserver started at http://127.23.214.130:43097/ using document root <none> and password file <none>
I20260923 17:19:26.603811 26359 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:26.603912 26359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:26.604176 26359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:26.606621 26359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "3932029473294f2da8133390f1a8dbee"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "427f48de8ab6492fb4635fc0996e3f9b"
server_key_iv: "9bb16a37bbd2b5798d3078e8dc923675"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.607275 26359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "3932029473294f2da8133390f1a8dbee"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "427f48de8ab6492fb4635fc0996e3f9b"
server_key_iv: "9bb16a37bbd2b5798d3078e8dc923675"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.612004 26359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:26.614817 26374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:26.616330 26359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:26.616483 26359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "3932029473294f2da8133390f1a8dbee"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "427f48de8ab6492fb4635fc0996e3f9b"
server_key_iv: "9bb16a37bbd2b5798d3078e8dc923675"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.616621 26359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:26.641067 26359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:26.644626 26359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:26.644860 26359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:26.646656 26359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:26.646739 26359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:26.646807 26359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:26.646852 26359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:26.658447 26359 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:43029
I20260923 17:19:26.658521 26489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:43029 every 8 connection(s)
I20260923 17:19:26.659428 26359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:26.661657 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 26359
I20260923 17:19:26.661777 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:26.662016 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.647627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:50039 (local address 127.23.214.190:35007)
0923 17:19:26.647779 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.647783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.649027 (+ 1244us) server_negotiation.cc:407] Connection header received
0923 17:19:26.650154 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.650158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.650280 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.650382 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.652071 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.652983 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.654590 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.654912 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.655032 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.657696 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.657700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.657703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.657738 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.659425 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.660135 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.660141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.660145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.660247 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.660660 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.660662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.660663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.660871 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.660956 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.661586 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.661762 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:19:26.662117 24410 external_mini_cluster.cc:1428] Setting key 685562f4a09c63059e4975eab34415b1
I20260923 17:19:26.662953 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.648301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35007 (local address 127.23.214.130:50039)
0923 17:19:26.648843 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.648919 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.649867 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.650594 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.650603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.651127 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:26.651867 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.651893 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.653138 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.653143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.654461 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.654471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.655095 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.655099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.655312 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.655959 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:26.655976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:26.657509 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:26.659580 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.659586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.659601 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.659995 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.660355 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.660358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.660360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.660539 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.660987 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:26.660993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:26.661282 (+ 289us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.661533 (+ 251us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.661951 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":149,"threads_started":1}
I20260923 17:19:26.664508 26490 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35007
I20260923 17:19:26.664673 26490 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:26.665315 26490 heartbeater.cc:511] Master 127.23.214.190:35007 requested a full tablet report, sending...
I20260923 17:19:26.666540 26146 ts_manager.cc:194] Registered new tserver with Master: 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029)
I20260923 17:19:26.667328 26146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:50039
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
I20260923 17:19:26.677783 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.667007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:54977 (local address 127.23.214.190:35007)
0923 17:19:26.667116 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.667120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.667386 (+ 266us) server_negotiation.cc:407] Connection header received
0923 17:19:26.670107 (+ 2721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.670111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.670164 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.670265 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.670884 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.671774 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.672658 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.673061 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.673199 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.674516 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.674519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.674521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.674547 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.675963 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.676407 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.676409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.676411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.676538 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.676842 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.676844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.676845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.677069 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.677190 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.677477 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.677589 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:19:26.677786 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.666941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35007 (local address 127.23.214.130:54977)
0923 17:19:26.667284 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.667299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.667412 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.670286 (+ 2874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.670289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.670537 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:26.670717 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.670725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.671925 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.671927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.672525 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.672534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.673207 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.673211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.673285 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.673987 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:26.673999 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:26.674350 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:26.676077 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.676080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.676082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.676282 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.676630 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.676636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.676640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.676745 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.677182 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:26.677185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:26.677299 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.677410 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.677578 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:26.693419 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35007
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:26.807682 26499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:26.808034 26499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:26.808116 26499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:26.812485 26499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:26.812584 26499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:26.812664 26499 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:19:26.812709 26499 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:26.812779 26499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:26.818349 26499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:26.819563 26499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:26.820703 26499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:26.831429 26504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:26.831429 26505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:26.831473 26499 server_base.cc:1034] running on GCE node
W20260923 17:19:26.831488 26507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:26.832119 26499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:26.832818 26499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:26.833983 26499 hybrid_clock.cc:648] HybridClock initialized: now 1790183966833943 us; error 53 us; skew 500 ppm
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:26.837095 26499 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:26.838261 26499 webserver.cc:460] Webserver started at http://127.23.214.131:39527/ using document root <none> and password file <none>
I20260923 17:19:26.838940 26499 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:26.839044 26499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:26.839385 26499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:26.841774 26499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "29b98c63ac7049119004395d1a2c1878"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "eaa368c8d69e1d57d20673ecd22bf4d2"
server_key_iv: "a960eadcad2aa49bd02ae693a2f26d9f"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.842478 26499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "29b98c63ac7049119004395d1a2c1878"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "eaa368c8d69e1d57d20673ecd22bf4d2"
server_key_iv: "a960eadcad2aa49bd02ae693a2f26d9f"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.846889 26499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:19:26.849591 26514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:26.851045 26499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:26.851200 26499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "29b98c63ac7049119004395d1a2c1878"
format_stamp: "Formatted at 2026-09-23 17:19:26 on dist-test-slave-9q7f"
server_key: "eaa368c8d69e1d57d20673ecd22bf4d2"
server_key_iv: "a960eadcad2aa49bd02ae693a2f26d9f"
server_key_version: "encryptionkey@0"
I20260923 17:19:26.851336 26499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:26.863205 26499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:26.866724 26499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:26.866956 26499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:26.868522 26499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:26.868602 26499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:26.868665 26499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:26.868716 26499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:26.880481 26499 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:34547
I20260923 17:19:26.880599 26629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:34547 every 8 connection(s)
I20260923 17:19:26.881454 26499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:26.884650 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.869462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:34647 (local address 127.23.214.190:35007)
0923 17:19:26.869599 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.869603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.870802 (+ 1199us) server_negotiation.cc:407] Connection header received
0923 17:19:26.872001 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.872006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.872071 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.872147 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.874081 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.874842 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.876126 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.876473 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.876590 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.880581 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.880585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.880587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.880623 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.882441 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.882987 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.882990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.882992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.883085 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.883444 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.883447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.883449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.883656 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.883796 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.884339 (+ 543us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.884471 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:19:26.885517 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.870120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35007 (local address 127.23.214.131:34647)
0923 17:19:26.870637 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.870684 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.871714 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.872674 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.872684 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.873216 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:26.873861 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.873933 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.874957 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.874961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.875921 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.875959 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.876667 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.876675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.876952 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.878477 (+ 1525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:26.878504 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:26.880386 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:26.882569 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.882575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.882589 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.882868 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.883181 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.883185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.883187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.883338 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.883824 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:26.883830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:26.884083 (+ 253us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.884284 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.884618 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":141,"threads_started":1}
I20260923 17:19:26.886767 26630 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35007
I20260923 17:19:26.886933 26630 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:26.887462 26630 heartbeater.cc:511] Master 127.23.214.190:35007 requested a full tablet report, sending...
I20260923 17:19:26.888532 26147 ts_manager.cc:194] Registered new tserver with Master: 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547)
I20260923 17:19:26.889330 26147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:34647
I20260923 17:19:26.890614 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 26499
I20260923 17:19:26.890760 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:26.891099 24410 external_mini_cluster.cc:1428] Setting key c08942e2fcb4377df82c59c6f801def8
I20260923 17:19:26.893222 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:19:26.897395 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.889090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:33803 (local address 127.23.214.190:35007)
0923 17:19:26.889219 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.889225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.889390 (+ 165us) server_negotiation.cc:407] Connection header received
0923 17:19:26.889523 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.889525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.889564 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.889651 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.890210 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.890867 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.891869 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.892205 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.892314 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.893627 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.893630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.893632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.893654 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.895612 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.896072 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.896077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.896080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.896177 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.896539 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.896541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.896542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.896722 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.896812 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.897159 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.897262 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:19:26.897593 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.889027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35007 (local address 127.23.214.131:33803)
0923 17:19:26.889300 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.889316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.889425 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.889688 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.889691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.889973 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:26.890093 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.890100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.890998 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.891001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.891745 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.891754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.892342 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.892346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.892446 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.893022 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:26.893037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:26.893490 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:26.895739 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.895742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.895743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.895971 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.896315 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:26.896318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:26.896322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:26.896427 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:26.896832 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:26.896838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:26.896969 (+ 131us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.897097 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.897295 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:19:26 dist-test-slave-9q7f krb5kdc[26102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:26.922199 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.912686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.23.214.190:35007)
0923 17:19:26.912886 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.912890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.912909 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:19:26.912957 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.912960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.913003 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.913094 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:26.913967 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.914536 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.915424 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.915645 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.915736 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.918724 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:26.918728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:26.918730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:26.918773 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:26.920497 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.920969 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.920972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.920974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.921054 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:26.921363 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:26.921366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:26.921367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:26.921565 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:26.921667 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.921960 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.922070 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 17:19:26.925799 26147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:26.929028 26147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:26.945276 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.938637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37057 (local address 127.0.0.1:49824)
0923 17:19:26.939075 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.939102 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.940542 (+ 1440us) spinlock_profiling.cc:243] Waited 938 us on lock 0x2c69dc88. stack: 00007fc6f749cc78 00007fc6f749d1a9 00007fc6f705f1d7 00000000005984a6 00000000005984f9 000000000059c488 00007fc6f74992ab 00007fc6f74989da 00007fc6ffbf6817 00007fc6ffc0907f 00007fc6ffbff003 00007fc6fbaf1267 00007fc6fbaef819 00007fc6fbb30832 00007fc6fbb3168c 00007fc6fbb470d3
0923 17:19:26.940563 (+ 21us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.941361 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.941364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.941382 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:26.941476 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.941481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.942732 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.942736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.943890 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.943901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.944792 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.944796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.944905 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.944928 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.945009 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.945085 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":30,"spinlock_wait_cycles":2156928,"thread_start_us":118,"threads_started":1}
I20260923 17:19:26.945278 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.939422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.23.214.129:37057)
0923 17:19:26.940821 (+ 1399us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.940827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.940850 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:19:26.940955 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.940962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.941156 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.941358 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:26.941597 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.942546 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.944017 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.944675 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.944793 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.944927 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.945040 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.945147 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300,"thread_start_us":78,"threads_started":1}
I20260923 17:19:26.945616 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.938655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.23.214.130:43029)
0923 17:19:26.939365 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.939378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.939402 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:19:26.939476 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.939503 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.939742 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.940358 (+ 616us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:26.940513 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.941722 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.943540 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.944418 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.944568 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.944719 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.945164 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.945252 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":137,"threads_started":1}
I20260923 17:19:26.945595 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.938475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:43029 (local address 127.0.0.1:46700)
0923 17:19:26.939039 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.939092 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.939259 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.939957 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.939965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.939989 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:26.940174 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.940182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.941832 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.941836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.943412 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.943421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.944557 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.944561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.944683 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.944748 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.944871 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.945021 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":62,"thread_start_us":101,"threads_started":1}
I20260923 17:19:26.946012 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.939363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34547 (local address 127.0.0.1:55640)
0923 17:19:26.939611 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:26.939624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:26.939699 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:26.940744 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:26.940747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:26.940766 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:26.940858 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.940864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.942236 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.942240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.943410 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:26.943421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.945163 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.945167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:26.945246 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.945266 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:19:26.945337 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:19:26.945857 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":34,"thread_start_us":110,"threads_started":1}
I20260923 17:19:26.946306 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:26.939781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.23.214.131:34547)
0923 17:19:26.940178 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:26.940189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:26.940281 (+ 92us) server_negotiation.cc:407] Connection header received
0923 17:19:26.940373 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:26.940380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:26.940598 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:26.940752 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:26.940981 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.942116 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.944059 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:26.944827 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:26.944964 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:26.945094 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:26.945539 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:19:26.945651 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":106,"threads_started":1}
I20260923 17:19:26.948841 26284 tablet_service.cc:1467] Processing CreateTablet for tablet ea5697ad4cac43ebb28930816cac4ca8 (DEFAULT_TABLE table=test-table [id=f2391fd35dee4d089eb038ea76597c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:26.948849 26414 tablet_service.cc:1467] Processing CreateTablet for tablet ea5697ad4cac43ebb28930816cac4ca8 (DEFAULT_TABLE table=test-table [id=f2391fd35dee4d089eb038ea76597c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:26.948848 26564 tablet_service.cc:1467] Processing CreateTablet for tablet ea5697ad4cac43ebb28930816cac4ca8 (DEFAULT_TABLE table=test-table [id=f2391fd35dee4d089eb038ea76597c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:26.950021 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5697ad4cac43ebb28930816cac4ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:26.950042 26414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5697ad4cac43ebb28930816cac4ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:26.950037 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5697ad4cac43ebb28930816cac4ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:26.957396 26653 tablet_bootstrap.cc:492] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Bootstrap starting.
I20260923 17:19:26.959092 26654 tablet_bootstrap.cc:492] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Bootstrap starting.
I20260923 17:19:26.959578 26655 tablet_bootstrap.cc:492] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Bootstrap starting.
I20260923 17:19:26.960005 26653 tablet_bootstrap.cc:654] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:26.960769 26653 log.cc:824] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:26.961411 26654 tablet_bootstrap.cc:654] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:26.961483 26655 tablet_bootstrap.cc:654] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:26.962332 26654 log.cc:824] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:26.962323 26655 log.cc:824] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:26.963615 26653 tablet_bootstrap.cc:492] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: No bootstrap required, opened a new log
I20260923 17:19:26.963836 26653 ts_tablet_manager.cc:1397] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:19:26.964473 26654 tablet_bootstrap.cc:492] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: No bootstrap required, opened a new log
I20260923 17:19:26.964610 26655 tablet_bootstrap.cc:492] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: No bootstrap required, opened a new log
I20260923 17:19:26.964658 26654 ts_tablet_manager.cc:1397] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:19:26.964749 26655 ts_tablet_manager.cc:1397] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:19:26.967916 26655 raft_consensus.cc:348] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:26.968204 26655 raft_consensus.cc:374] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:26.968271 26655 raft_consensus.cc:729] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3932029473294f2da8133390f1a8dbee, State: Initialized, Role: FOLLOWER
I20260923 17:19:26.968746 26655 consensus_queue.cc:260] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:26.968648 26653 raft_consensus.cc:348] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:26.968648 26654 raft_consensus.cc:348] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:26.968919 26654 raft_consensus.cc:374] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:26.968919 26653 raft_consensus.cc:374] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:26.968981 26653 raft_consensus.cc:729] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc811c7cf7a44433b7e3e601d5ec5702, State: Initialized, Role: FOLLOWER
I20260923 17:19:26.968981 26654 raft_consensus.cc:729] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Initialized, Role: FOLLOWER
I20260923 17:19:26.969436 26654 consensus_queue.cc:260] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:26.969436 26653 consensus_queue.cc:260] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:26.970520 26653 ts_tablet_manager.cc:1428] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 17:19:26.970520 26654 ts_tablet_manager.cc:1428] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:19:26.970520 26655 ts_tablet_manager.cc:1428] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:19:26.970892 26350 heartbeater.cc:503] Master 127.23.214.190:35007 was elected leader, sending a full tablet report...
I20260923 17:19:26.970924 26630 heartbeater.cc:503] Master 127.23.214.190:35007 was elected leader, sending a full tablet report...
I20260923 17:19:26.971956 26490 heartbeater.cc:503] Master 127.23.214.190:35007 was elected leader, sending a full tablet report...
W20260923 17:19:27.132563 26631 tablet.cc:2369] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:19:27.160779 26491 tablet.cc:2369] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:19:27.168812 26351 tablet.cc:2369] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:27.196409 26661 raft_consensus.cc:482] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:27.196688 26661 raft_consensus.cc:504] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.198032 26661 leader_election.cc:288] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029), 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547)
I20260923 17:19:27.202082 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.198393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:33829 (local address 127.23.214.131:34547)
0923 17:19:27.198522 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.198527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.198717 (+ 190us) server_negotiation.cc:407] Connection header received
0923 17:19:27.198897 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.198899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.198941 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.199051 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:27.199358 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.199864 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.201017 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.201457 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.201590 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.201644 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.201852 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.201948 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:19:27.202077 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.198394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:41261 (local address 127.23.214.130:43029)
0923 17:19:27.198592 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.198597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.198724 (+ 127us) server_negotiation.cc:407] Connection header received
0923 17:19:27.198914 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.198917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.198967 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.199051 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:27.199316 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.199904 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.201005 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.201431 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.201571 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.201625 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.201841 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.201937 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260923 17:19:27.202138 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.198239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:43029 (local address 127.23.214.129:41261)
0923 17:19:27.198626 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.198648 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.198800 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.199064 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.199067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.199082 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:27.199212 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.199217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.200020 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.200024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.200884 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.200893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.201567 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.201571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.201666 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.201730 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.201818 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.201925 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":114,"spinlock_wait_cycles":37120,"thread_start_us":99,"threads_started":1}
I20260923 17:19:27.202142 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.198239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34547 (local address 127.23.214.129:33829)
0923 17:19:27.198607 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.198619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.198766 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.199070 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.199073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.199098 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:27.199246 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.199251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.199984 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.199987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.200895 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.200904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.201579 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.201591 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.201711 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.201748 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.201837 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.201952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":50,"spinlock_wait_cycles":34944,"thread_start_us":122,"threads_started":1}
I20260923 17:19:27.202886 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5697ad4cac43ebb28930816cac4ca8" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee" is_pre_election: true
I20260923 17:19:27.202942 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5697ad4cac43ebb28930816cac4ca8" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b98c63ac7049119004395d1a2c1878" is_pre_election: true
I20260923 17:19:27.203235 26568 raft_consensus.cc:2454] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 0.
I20260923 17:19:27.203235 26444 raft_consensus.cc:2454] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 0.
I20260923 17:19:27.203774 26237 leader_election.cc:302] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, 3932029473294f2da8133390f1a8dbee, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:27.204133 26661 raft_consensus.cc:2781] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:27.204274 26661 raft_consensus.cc:482] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:27.204347 26661 raft_consensus.cc:3037] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.205513 26661 raft_consensus.cc:504] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.205984 26661 leader_election.cc:288] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 election: Requested vote from peers 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029), 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547)
I20260923 17:19:27.206359 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5697ad4cac43ebb28930816cac4ca8" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee"
I20260923 17:19:27.206385 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5697ad4cac43ebb28930816cac4ca8" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b98c63ac7049119004395d1a2c1878"
I20260923 17:19:27.206506 26444 raft_consensus.cc:3037] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.206532 26568 raft_consensus.cc:3037] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.207662 26444 raft_consensus.cc:2454] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 1.
I20260923 17:19:27.207942 26568 raft_consensus.cc:2454] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 1.
I20260923 17:19:27.208030 26237 leader_election.cc:302] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3932029473294f2da8133390f1a8dbee, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:27.208246 26661 raft_consensus.cc:2781] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:27.208500 26661 raft_consensus.cc:686] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 LEADER]: Becoming Leader. State: Replica: dc811c7cf7a44433b7e3e601d5ec5702, State: Running, Role: LEADER
I20260923 17:19:27.208884 26661 consensus_queue.cc:237] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.212466 26147 catalog_manager.cc:5495] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129). New cstate: current_term: 1 leader_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:19:27 dist-test-slave-9q7f krb5kdc[26102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:27.269335 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.259900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.23.214.190:35007)
0923 17:19:27.260083 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.260088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.260287 (+ 199us) server_negotiation.cc:407] Connection header received
0923 17:19:27.260336 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.260338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.260382 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.260458 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:27.261135 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.261765 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.262583 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.262821 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.262911 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.265395 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:27.265400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:27.265405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:27.265446 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:27.267084 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:27.267753 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:27.267756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:27.267758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:27.267854 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:27.268193 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:27.268200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:27.268202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:27.268397 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:27.268504 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.268906 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.269133 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":87}
I20260923 17:19:27.269337 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.259772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35007 (local address 127.0.0.1:39802)
0923 17:19:27.260099 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.260112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.260222 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.260495 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.260497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.260914 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:27.261006 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.261012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.261902 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.261904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.262468 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.262475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.262937 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.262940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.263004 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.263584 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:27.263596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:27.265163 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:27.267195 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:27.267208 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:27.267218 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:27.267622 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:27.267959 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:27.267964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:27.267969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:27.268077 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:27.268512 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:27.268521 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:27.268672 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.268815 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.269013 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260923 17:19:27.273808 26147 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:39802:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:19:27.274719 26147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:27.279801 26564 tablet_service.cc:1467] Processing CreateTablet for tablet f5c72c7ba1f342dd9b98d3de57013b27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e76c5ea052049cd8d2247dcef8799c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:27.279670 26414 tablet_service.cc:1467] Processing CreateTablet for tablet f5c72c7ba1f342dd9b98d3de57013b27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e76c5ea052049cd8d2247dcef8799c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:27.279670 26284 tablet_service.cc:1467] Processing CreateTablet for tablet f5c72c7ba1f342dd9b98d3de57013b27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e76c5ea052049cd8d2247dcef8799c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:27.280475 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c72c7ba1f342dd9b98d3de57013b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:27.280475 26414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c72c7ba1f342dd9b98d3de57013b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:27.280481 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c72c7ba1f342dd9b98d3de57013b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:27.283061 26654 tablet_bootstrap.cc:492] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: Bootstrap starting.
I20260923 17:19:27.283352 26653 tablet_bootstrap.cc:492] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: Bootstrap starting.
I20260923 17:19:27.283494 26655 tablet_bootstrap.cc:492] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: Bootstrap starting.
I20260923 17:19:27.284092 26654 tablet_bootstrap.cc:654] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:27.284338 26653 tablet_bootstrap.cc:654] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:27.284420 26655 tablet_bootstrap.cc:654] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:27.285699 26655 tablet_bootstrap.cc:492] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: No bootstrap required, opened a new log
I20260923 17:19:27.285763 26654 tablet_bootstrap.cc:492] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: No bootstrap required, opened a new log
I20260923 17:19:27.285790 26655 ts_tablet_manager.cc:1397] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:27.285851 26654 ts_tablet_manager.cc:1397] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:27.286124 26653 tablet_bootstrap.cc:492] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: No bootstrap required, opened a new log
I20260923 17:19:27.286211 26653 ts_tablet_manager.cc:1397] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:27.286324 26655 raft_consensus.cc:348] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.286446 26655 raft_consensus.cc:374] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:27.286479 26655 raft_consensus.cc:729] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3932029473294f2da8133390f1a8dbee, State: Initialized, Role: FOLLOWER
I20260923 17:19:27.286425 26654 raft_consensus.cc:348] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.286548 26654 raft_consensus.cc:374] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:27.286590 26654 raft_consensus.cc:729] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Initialized, Role: FOLLOWER
I20260923 17:19:27.286621 26655 consensus_queue.cc:260] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.286705 26654 consensus_queue.cc:260] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.286724 26653 raft_consensus.cc:348] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.286821 26653 raft_consensus.cc:374] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:27.286861 26653 raft_consensus.cc:729] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc811c7cf7a44433b7e3e601d5ec5702, State: Initialized, Role: FOLLOWER
I20260923 17:19:27.286922 26655 ts_tablet_manager.cc:1428] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:27.287004 26654 ts_tablet_manager.cc:1428] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:27.286978 26653 consensus_queue.cc:260] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.287032 26660 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.287160 26660 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:27.287245 26653 ts_tablet_manager.cc:1428] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:27.287335 26659 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.287482 26659 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:27.287889 26661 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.288033 26661 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:27.492913 26660 raft_consensus.cc:482] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:27.493065 26660 raft_consensus.cc:504] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.494381 26660 leader_election.cc:288] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057), 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029)
I20260923 17:19:27.498549 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.494779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:36555 (local address 127.23.214.130:43029)
0923 17:19:27.494959 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.494966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.495192 (+ 226us) server_negotiation.cc:407] Connection header received
0923 17:19:27.495315 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.495318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.495369 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.495450 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:27.495752 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.496232 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.497434 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.497837 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.498070 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.498127 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.498336 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.498408 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:19:27.498565 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.494581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37057 (local address 127.23.214.131:40899)
0923 17:19:27.494942 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.494958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.495070 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.495358 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.495362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.495378 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:27.495531 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.495538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.496369 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.496372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.497308 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.497326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.498073 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.498077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.498204 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.498222 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.498321 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.498408 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":96,"thread_start_us":86,"threads_started":1}
I20260923 17:19:27.498558 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.494694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:40899 (local address 127.23.214.129:37057)
0923 17:19:27.495083 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.495090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.495109 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:19:27.495175 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.495178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.495230 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.495326 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:27.495654 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.496221 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.497465 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.497864 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.498071 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.498127 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.498332 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.498414 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":83,"threads_started":1}
I20260923 17:19:27.498570 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.494711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:43029 (local address 127.23.214.131:36555)
0923 17:19:27.495062 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.495077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.495194 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.495475 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.495477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.495492 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:27.495642 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.495648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.496370 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.496373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.497309 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.497318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.498063 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.498067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.498208 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.498225 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.498318 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.498418 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":121,"thread_start_us":66,"threads_started":1}
I20260923 17:19:27.499055 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c72c7ba1f342dd9b98d3de57013b27" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee" is_pre_election: true
I20260923 17:19:27.499244 26444 raft_consensus.cc:2454] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 0.
I20260923 17:19:27.499570 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c72c7ba1f342dd9b98d3de57013b27" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" is_pre_election: true
I20260923 17:19:27.499614 26517 leader_election.cc:302] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, 3932029473294f2da8133390f1a8dbee; no voters:
I20260923 17:19:27.499790 26304 raft_consensus.cc:2454] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 0.
I20260923 17:19:27.499913 26660 raft_consensus.cc:2781] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:27.499962 26660 raft_consensus.cc:482] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:27.499989 26660 raft_consensus.cc:3037] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.500988 26660 raft_consensus.cc:504] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.501349 26660 leader_election.cc:288] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 election: Requested vote from peers dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057), 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029)
I20260923 17:19:27.501693 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c72c7ba1f342dd9b98d3de57013b27" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee"
I20260923 17:19:27.501695 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c72c7ba1f342dd9b98d3de57013b27" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc811c7cf7a44433b7e3e601d5ec5702"
I20260923 17:19:27.501791 26444 raft_consensus.cc:3037] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.501823 26304 raft_consensus.cc:3037] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.502691 26444 raft_consensus.cc:2454] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 1.
I20260923 17:19:27.502815 26304 raft_consensus.cc:2454] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 1.
I20260923 17:19:27.502992 26517 leader_election.cc:302] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, 3932029473294f2da8133390f1a8dbee; no voters:
I20260923 17:19:27.503182 26660 raft_consensus.cc:2781] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:27.503427 26660 raft_consensus.cc:686] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [term 1 LEADER]: Becoming Leader. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Running, Role: LEADER
I20260923 17:19:27.504065 26660 consensus_queue.cc:237] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.507261 26678 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: TxnStatusTablet state changed. Reason: New leader 29b98c63ac7049119004395d1a2c1878. Latest consensus state: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.507383 26678 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:27.507687 26680 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:19:27.507915 26680 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:19:27.507870 26147 catalog_manager.cc:5495] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b98c63ac7049119004395d1a2c1878 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } health_report { overall_health: HEALTHY } } }
I20260923 17:19:27.529091 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.525246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.23.214.131:34547)
0923 17:19:27.525369 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.525373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.525386 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:19:27.525563 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.525565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.525618 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.525700 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:27.525971 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.526483 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.527690 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.527896 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.527972 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.528296 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.528730 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.528820 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.528925 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.528978 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 17:19:27.529152 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.525098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34547 (local address 127.0.0.1:55650)
0923 17:19:27.525301 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.525318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.525424 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.525734 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.525738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.525745 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:27.525858 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.525864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.526587 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.526591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.527568 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.527578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.528001 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.528005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.528101 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.528158 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.528823 (+ 665us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.528837 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.528909 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.529002 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:19:27.535441 26444 raft_consensus.cc:1261] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Refusing update from remote peer 29b98c63ac7049119004395d1a2c1878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:27.535440 26304 raft_consensus.cc:1261] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Refusing update from remote peer 29b98c63ac7049119004395d1a2c1878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:27.536113 26660 consensus_queue.cc:1035] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:27.536324 26678 consensus_queue.cc:1035] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:27.539826 26661 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: TxnStatusTablet state changed. Reason: New leader 29b98c63ac7049119004395d1a2c1878. Latest consensus state: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.539965 26661 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:27.540411 26659 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: TxnStatusTablet state changed. Reason: New leader 29b98c63ac7049119004395d1a2c1878. Latest consensus state: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.540522 26659 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:27.544083 26660 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.544217 26660 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:27.544349 26660 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.544421 26660 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P 29b98c63ac7049119004395d1a2c1878: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:27.544617 26661 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.544716 26661 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P dc811c7cf7a44433b7e3e601d5ec5702: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:27.545212 26685 mvcc.cc:205] Tried to move back new op lower bound from 7332593531017338880 to 7332593530899869696. Current Snapshot: MvccSnapshot[applied={T|T < 7332593531017338880}]
I20260923 17:19:27.546275 26659 tablet_replica.cc:439] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } } }
I20260923 17:19:27.546404 26659 tablet_replica.cc:442] T f5c72c7ba1f342dd9b98d3de57013b27 P 3932029473294f2da8133390f1a8dbee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:27.556087 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.551448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.23.214.129:37057)
0923 17:19:27.551570 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.551577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.551767 (+ 190us) server_negotiation.cc:407] Connection header received
0923 17:19:27.551960 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.551963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.552022 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.552127 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:27.552389 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.553006 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.554380 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.554691 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.554783 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.555088 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.555632 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.555726 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.555830 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.555889 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:19:27.568590 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.564666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34547 (local address 127.23.214.129:57627)
0923 17:19:27.565033 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.565046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.565144 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.565387 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.565389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.565394 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:27.565575 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.565581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.566612 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.566615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.567250 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.567258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.567846 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.567849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.567934 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.567984 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.568327 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.568332 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.568381 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.568447 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
I20260923 17:19:27.568583 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.564763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:57627 (local address 127.23.214.131:34547)
0923 17:19:27.564911 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.564916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.565118 (+ 202us) server_negotiation.cc:407] Connection header received
0923 17:19:27.565224 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.565227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.565289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.565373 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:27.565679 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.566502 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.567366 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.567712 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.567818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.568148 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.568233 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.568312 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.568393 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.568445 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:19:27.576483 26568 raft_consensus.cc:1261] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Refusing update from remote peer dc811c7cf7a44433b7e3e601d5ec5702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:27.576603 26444 raft_consensus.cc:1261] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Refusing update from remote peer dc811c7cf7a44433b7e3e601d5ec5702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:27.577080 26664 consensus_queue.cc:1035] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:27.577270 26661 consensus_queue.cc:1035] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:27.582993 26684 mvcc.cc:205] Tried to move back new op lower bound from 7332593531187589120 to 7332593529690705920. Current Snapshot: MvccSnapshot[applied={T|T < 7332593531187589120}]
I20260923 17:19:27.583801 26683 mvcc.cc:205] Tried to move back new op lower bound from 7332593531187589120 to 7332593529690705920. Current Snapshot: MvccSnapshot[applied={T|T < 7332593531187589120}]
I20260923 17:19:27.603024 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.599112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:32823 (local address 127.23.214.129:37057)
0923 17:19:27.599250 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.599257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.599499 (+ 242us) server_negotiation.cc:407] Connection header received
0923 17:19:27.599659 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.599662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.599713 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.599801 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:27.600111 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.600770 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.601944 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.602169 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.602284 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.602579 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.602656 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.602735 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.602827 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.602880 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:19:27.603041 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.599025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37057 (local address 127.23.214.131:32823)
0923 17:19:27.599417 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.599429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.599548 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.599809 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.599812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.599834 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:27.599976 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.599982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.601078 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.601081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.601793 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.601802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.602281 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.602285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.602389 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.602443 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.602738 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.602747 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.602812 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.602889 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260923 17:19:27.634377 26147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:27.634594 26147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:27.637625 26147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Sending DeleteTablet for 3 replicas of tablet ea5697ad4cac43ebb28930816cac4ca8
I20260923 17:19:27.638741 26414 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5697ad4cac43ebb28930816cac4ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:27 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:46700
I20260923 17:19:27.638911 26284 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5697ad4cac43ebb28930816cac4ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:27 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:49824
I20260923 17:19:27.639780 26705 tablet_replica.cc:330] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: stopping tablet replica
I20260923 17:19:27.639955 26564 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5697ad4cac43ebb28930816cac4ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:27 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:55640
I20260923 17:19:27.640290 26705 raft_consensus.cc:2229] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:27.640561 26705 raft_consensus.cc:2258] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:27.640769 26147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:27.641247 26147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:27.642452 26705 ts_tablet_manager.cc:1905] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:27.645862 26705 ts_tablet_manager.cc:1918] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:27.646101 26705 log.cc:1196] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/ea5697ad4cac43ebb28930816cac4ca8
I20260923 17:19:27.646538 26705 ts_tablet_manager.cc:1939] T ea5697ad4cac43ebb28930816cac4ca8 P 3932029473294f2da8133390f1a8dbee: Deleting consensus metadata
I20260923 17:19:27.646909 26414 tablet_service.cc:1467] Processing CreateTablet for tablet 16352efd614d4d18982569d7544ba39e (DEFAULT_TABLE table=test-table [id=9ed63c53f14343e982ee74626052e5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:27.647190 26414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16352efd614d4d18982569d7544ba39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:27.648157 26133 catalog_manager.cc:4841] TS 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029): tablet ea5697ad4cac43ebb28930816cac4ca8 (table test-table [id=f2391fd35dee4d089eb038ea76597c86]) successfully deleted
I20260923 17:19:27.649490 26655 tablet_bootstrap.cc:492] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: Bootstrap starting.
I20260923 17:19:27.650101 26707 tablet_replica.cc:330] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: stopping tablet replica
I20260923 17:19:27.650348 26284 tablet_service.cc:1467] Processing CreateTablet for tablet 16352efd614d4d18982569d7544ba39e (DEFAULT_TABLE table=test-table [id=9ed63c53f14343e982ee74626052e5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:27.650457 26707 raft_consensus.cc:2229] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:27.650606 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16352efd614d4d18982569d7544ba39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:27.650643 26655 tablet_bootstrap.cc:654] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:27.650652 26707 raft_consensus.cc:2258] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:27.652769 26707 ts_tablet_manager.cc:1905] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:27.653158 26564 tablet_service.cc:1467] Processing CreateTablet for tablet 16352efd614d4d18982569d7544ba39e (DEFAULT_TABLE table=test-table [id=9ed63c53f14343e982ee74626052e5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:27.653448 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16352efd614d4d18982569d7544ba39e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:27.656081 26654 tablet_bootstrap.cc:492] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: Bootstrap starting.
I20260923 17:19:27.656490 26707 ts_tablet_manager.cc:1918] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:27.656629 26653 tablet_bootstrap.cc:492] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: Bootstrap starting.
I20260923 17:19:27.656656 26707 log.cc:1196] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/ea5697ad4cac43ebb28930816cac4ca8
I20260923 17:19:27.657150 26707 ts_tablet_manager.cc:1939] T ea5697ad4cac43ebb28930816cac4ca8 P 29b98c63ac7049119004395d1a2c1878: Deleting consensus metadata
I20260923 17:19:27.657606 26654 tablet_bootstrap.cc:654] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:27.657624 26653 tablet_bootstrap.cc:654] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:27.658785 26132 catalog_manager.cc:4841] TS 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547): tablet ea5697ad4cac43ebb28930816cac4ca8 (table test-table [id=f2391fd35dee4d089eb038ea76597c86]) successfully deleted
I20260923 17:19:27.663551 26654 tablet_bootstrap.cc:492] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: No bootstrap required, opened a new log
I20260923 17:19:27.663663 26654 ts_tablet_manager.cc:1397] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260923 17:19:27.664181 26654 raft_consensus.cc:348] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.664310 26654 raft_consensus.cc:374] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:27.664346 26654 raft_consensus.cc:729] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Initialized, Role: FOLLOWER
I20260923 17:19:27.664482 26654 consensus_queue.cc:260] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.664831 26654 ts_tablet_manager.cc:1428] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:27.667210 26655 tablet_bootstrap.cc:492] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: No bootstrap required, opened a new log
I20260923 17:19:27.667322 26655 ts_tablet_manager.cc:1397] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20260923 17:19:27.667501 26653 tablet_bootstrap.cc:492] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: No bootstrap required, opened a new log
I20260923 17:19:27.667586 26653 ts_tablet_manager.cc:1397] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260923 17:19:27.667819 26655 raft_consensus.cc:348] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.667944 26655 raft_consensus.cc:374] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:27.667979 26655 raft_consensus.cc:729] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3932029473294f2da8133390f1a8dbee, State: Initialized, Role: FOLLOWER
I20260923 17:19:27.668036 26653 raft_consensus.cc:348] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.668147 26653 raft_consensus.cc:374] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:27.668115 26655 consensus_queue.cc:260] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.668179 26653 raft_consensus.cc:729] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc811c7cf7a44433b7e3e601d5ec5702, State: Initialized, Role: FOLLOWER
I20260923 17:19:27.668291 26653 consensus_queue.cc:260] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.668394 26655 ts_tablet_manager.cc:1428] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:27.668555 26653 ts_tablet_manager.cc:1428] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:27.672629 26706 tablet_replica.cc:330] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: stopping tablet replica
I20260923 17:19:27.673123 26706 raft_consensus.cc:2229] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:27.673636 26706 raft_consensus.cc:2258] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:27.675208 26706 ts_tablet_manager.cc:1905] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:27.678824 26706 ts_tablet_manager.cc:1918] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:27.679013 26706 log.cc:1196] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/ea5697ad4cac43ebb28930816cac4ca8
I20260923 17:19:27.679392 26706 ts_tablet_manager.cc:1939] T ea5697ad4cac43ebb28930816cac4ca8 P dc811c7cf7a44433b7e3e601d5ec5702: Deleting consensus metadata
I20260923 17:19:27.680694 26132 catalog_manager.cc:4841] TS dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057): tablet ea5697ad4cac43ebb28930816cac4ca8 (table test-table [id=f2391fd35dee4d089eb038ea76597c86]) successfully deleted
I20260923 17:19:27.854748 26678 raft_consensus.cc:482] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:27.854899 26678 raft_consensus.cc:504] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.855355 26678 leader_election.cc:288] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057), 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029)
I20260923 17:19:27.855963 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16352efd614d4d18982569d7544ba39e" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee" is_pre_election: true
I20260923 17:19:27.855963 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16352efd614d4d18982569d7544ba39e" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" is_pre_election: true
I20260923 17:19:27.856129 26304 raft_consensus.cc:2454] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 0.
I20260923 17:19:27.856129 26444 raft_consensus.cc:2454] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 0.
I20260923 17:19:27.856460 26517 leader_election.cc:302] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, 3932029473294f2da8133390f1a8dbee, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:27.856696 26678 raft_consensus.cc:2781] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:27.856743 26678 raft_consensus.cc:482] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:27.856770 26678 raft_consensus.cc:3037] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.857833 26678 raft_consensus.cc:504] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.858230 26678 leader_election.cc:288] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 election: Requested vote from peers dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057), 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029)
I20260923 17:19:27.858644 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16352efd614d4d18982569d7544ba39e" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc811c7cf7a44433b7e3e601d5ec5702"
I20260923 17:19:27.858786 26304 raft_consensus.cc:3037] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.858800 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16352efd614d4d18982569d7544ba39e" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee"
I20260923 17:19:27.859000 26444 raft_consensus.cc:3037] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:27.859692 26304 raft_consensus.cc:2454] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 1.
I20260923 17:19:27.860019 26516 leader_election.cc:302] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:27.860045 26444 raft_consensus.cc:2454] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 1.
I20260923 17:19:27.860185 26678 raft_consensus.cc:2781] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:27.860291 26678 raft_consensus.cc:686] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 1 LEADER]: Becoming Leader. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Running, Role: LEADER
I20260923 17:19:27.860438 26678 consensus_queue.cc:237] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:27.862408 26146 catalog_manager.cc:5495] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b98c63ac7049119004395d1a2c1878 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } health_report { overall_health: HEALTHY } } }
I20260923 17:19:27.921489 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.918624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.23.214.131:34547)
0923 17:19:27.918749 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.918753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.918764 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:19:27.918802 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.918804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.918851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.918939 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:27.919114 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.919609 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.920402 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.920633 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.920746 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.921037 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.921116 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.921196 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.921296 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.921344 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:19:27.930773 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.927583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34547 (local address 127.23.214.131:57219)
0923 17:19:27.927745 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:27.927761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:27.927877 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:27.928110 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:27.928113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:27.928119 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:27.928297 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.928308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.929012 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.929015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.929586 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:27.929593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.930105 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.930108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:27.930190 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.930210 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.930490 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.930499 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:19:27.930558 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 17:19:27.930640 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":37}
I20260923 17:19:27.930773 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:27.927739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:57219 (local address 127.23.214.131:34547)
0923 17:19:27.927864 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:27.927868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:27.927882 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:19:27.927988 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:27.927991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:27.928025 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:27.928105 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:27.928407 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.928900 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.929703 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:27.929990 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:27.930095 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:27.930327 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:27.930403 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:27.930478 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:27.930565 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 17:19:27.930609 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":64}
I20260923 17:19:27.937353 26304 raft_consensus.cc:1261] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Refusing update from remote peer 29b98c63ac7049119004395d1a2c1878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:27.937353 26444 raft_consensus.cc:1261] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Refusing update from remote peer 29b98c63ac7049119004395d1a2c1878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:27.937747 26703 consensus_queue.cc:1035] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:27.937949 26660 consensus_queue.cc:1035] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:19:27.952546 26683 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:19:29.509555 26145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:29.509747 26145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:29.513115 26145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Sending DeleteTablet for 3 replicas of tablet 16352efd614d4d18982569d7544ba39e
I20260923 17:19:29.513713 26284 tablet_service.cc:1514] Processing DeleteTablet for tablet 16352efd614d4d18982569d7544ba39e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:29 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:49824
I20260923 17:19:29.513990 26563 tablet_service.cc:1514] Processing DeleteTablet for tablet 16352efd614d4d18982569d7544ba39e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:29 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:55640
I20260923 17:19:29.514140 26414 tablet_service.cc:1514] Processing DeleteTablet for tablet 16352efd614d4d18982569d7544ba39e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:29 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:46700
I20260923 17:19:29.514364 26737 tablet_replica.cc:330] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: stopping tablet replica
I20260923 17:19:29.514436 26736 tablet_replica.cc:330] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: stopping tablet replica
I20260923 17:19:29.514525 26737 raft_consensus.cc:2229] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:29.514595 26736 raft_consensus.cc:2229] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:29.514729 26736 raft_consensus.cc:2258] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:29.516216 26736 ts_tablet_manager.cc:1905] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:29.518476 26737 raft_consensus.cc:2258] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:29.519264 26737 ts_tablet_manager.cc:1905] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:29.519538 26736 ts_tablet_manager.cc:1918] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:19:29.519651 26736 log.cc:1196] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/16352efd614d4d18982569d7544ba39e
I20260923 17:19:29.519970 26736 ts_tablet_manager.cc:1939] T 16352efd614d4d18982569d7544ba39e P dc811c7cf7a44433b7e3e601d5ec5702: Deleting consensus metadata
I20260923 17:19:29.520892 26738 tablet_replica.cc:330] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: stopping tablet replica
I20260923 17:19:29.520887 26132 catalog_manager.cc:4841] TS dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057): tablet 16352efd614d4d18982569d7544ba39e (table test-table [id=9ed63c53f14343e982ee74626052e5a6]) successfully deleted
I20260923 17:19:29.521106 26738 raft_consensus.cc:2229] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:29.521252 26738 raft_consensus.cc:2258] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:29.521996 26738 ts_tablet_manager.cc:1905] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:29.522603 26737 ts_tablet_manager.cc:1918] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:19:29.522712 26737 log.cc:1196] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/16352efd614d4d18982569d7544ba39e
I20260923 17:19:29.522979 26737 ts_tablet_manager.cc:1939] T 16352efd614d4d18982569d7544ba39e P 29b98c63ac7049119004395d1a2c1878: Deleting consensus metadata
I20260923 17:19:29.523746 26132 catalog_manager.cc:4841] TS 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547): tablet 16352efd614d4d18982569d7544ba39e (table test-table [id=9ed63c53f14343e982ee74626052e5a6]) successfully deleted
I20260923 17:19:29.525301 26738 ts_tablet_manager.cc:1918] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:19:29.525393 26738 log.cc:1196] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/16352efd614d4d18982569d7544ba39e
I20260923 17:19:29.525707 26738 ts_tablet_manager.cc:1939] T 16352efd614d4d18982569d7544ba39e P 3932029473294f2da8133390f1a8dbee: Deleting consensus metadata
I20260923 17:19:29.526762 26133 catalog_manager.cc:4841] TS 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029): tablet 16352efd614d4d18982569d7544ba39e (table test-table [id=9ed63c53f14343e982ee74626052e5a6]) successfully deleted
Sep 23 17:19:29 dist-test-slave-9q7f krb5kdc[26102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183969, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:19:29 dist-test-slave-9q7f krb5kdc[26102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183969, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:29.551860 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:29.535317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.23.214.190:35007)
0923 17:19:29.535606 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:29.535618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:29.535703 (+ 85us) server_negotiation.cc:407] Connection header received
0923 17:19:29.535885 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:29.535888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:29.535965 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:29.536061 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:29.536782 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:29.537356 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:29.538286 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:29.538530 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:29.538646 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:29.548504 (+ 9858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:29.548508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:29.548510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:29.548541 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:29.550016 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:29.550477 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:29.550481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:29.550483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:29.550591 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:29.550925 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:29.550928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:29.550936 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:29.551140 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:29.551288 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:29.551532 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:19:29.551665 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":124,"threads_started":1}
I20260923 17:19:29.555158 26145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:29.555598 26145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:29.560554 26414 tablet_service.cc:1467] Processing CreateTablet for tablet 0af3720844db48beb5a65a84f23335fb (DEFAULT_TABLE table=test-table [id=35fc99ddfb51485e933534f3a41b26ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:29.560552 26284 tablet_service.cc:1467] Processing CreateTablet for tablet 0af3720844db48beb5a65a84f23335fb (DEFAULT_TABLE table=test-table [id=35fc99ddfb51485e933534f3a41b26ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:29.560887 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af3720844db48beb5a65a84f23335fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:29.560889 26414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af3720844db48beb5a65a84f23335fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:29.560876 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 0af3720844db48beb5a65a84f23335fb (DEFAULT_TABLE table=test-table [id=35fc99ddfb51485e933534f3a41b26ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:29.561221 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af3720844db48beb5a65a84f23335fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:29.563398 26747 tablet_bootstrap.cc:492] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: Bootstrap starting.
I20260923 17:19:29.563721 26748 tablet_bootstrap.cc:492] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: Bootstrap starting.
I20260923 17:19:29.564414 26747 tablet_bootstrap.cc:654] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:29.565018 26748 tablet_bootstrap.cc:654] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:29.565513 26747 tablet_bootstrap.cc:492] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: No bootstrap required, opened a new log
I20260923 17:19:29.565618 26747 ts_tablet_manager.cc:1397] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:19:29.565833 26749 tablet_bootstrap.cc:492] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: Bootstrap starting.
I20260923 17:19:29.566133 26747 raft_consensus.cc:348] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.566273 26747 raft_consensus.cc:374] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:29.566309 26747 raft_consensus.cc:729] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Initialized, Role: FOLLOWER
I20260923 17:19:29.566484 26747 consensus_queue.cc:260] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.566843 26747 ts_tablet_manager.cc:1428] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:29.566859 26749 tablet_bootstrap.cc:654] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:29.567801 26748 tablet_bootstrap.cc:492] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: No bootstrap required, opened a new log
I20260923 17:19:29.567929 26748 ts_tablet_manager.cc:1397] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:19:29.568065 26749 tablet_bootstrap.cc:492] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: No bootstrap required, opened a new log
I20260923 17:19:29.568162 26749 ts_tablet_manager.cc:1397] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 17:19:29.568478 26748 raft_consensus.cc:348] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.568609 26748 raft_consensus.cc:374] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:29.568645 26748 raft_consensus.cc:729] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3932029473294f2da8133390f1a8dbee, State: Initialized, Role: FOLLOWER
I20260923 17:19:29.568585 26749 raft_consensus.cc:348] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.568734 26749 raft_consensus.cc:374] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:29.568766 26749 raft_consensus.cc:729] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc811c7cf7a44433b7e3e601d5ec5702, State: Initialized, Role: FOLLOWER
I20260923 17:19:29.568861 26748 consensus_queue.cc:260] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.568897 26749 consensus_queue.cc:260] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.569281 26748 ts_tablet_manager.cc:1428] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 17:19:29.569283 26749 ts_tablet_manager.cc:1428] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:19:29.852460 26731 raft_consensus.cc:482] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:29.852661 26731 raft_consensus.cc:504] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.853183 26731 leader_election.cc:288] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057), 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029)
I20260923 17:19:29.853772 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3720844db48beb5a65a84f23335fb" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee" is_pre_election: true
I20260923 17:19:29.853793 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3720844db48beb5a65a84f23335fb" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" is_pre_election: true
I20260923 17:19:29.853981 26304 raft_consensus.cc:2454] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 0.
I20260923 17:19:29.853986 26444 raft_consensus.cc:2454] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 0.
I20260923 17:19:29.854358 26516 leader_election.cc:302] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:29.854548 26731 raft_consensus.cc:2781] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:29.854614 26731 raft_consensus.cc:482] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:29.854679 26731 raft_consensus.cc:3037] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:29.855754 26731 raft_consensus.cc:504] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.856117 26731 leader_election.cc:288] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 election: Requested vote from peers dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057), 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029)
I20260923 17:19:29.856457 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3720844db48beb5a65a84f23335fb" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee"
I20260923 17:19:29.856457 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af3720844db48beb5a65a84f23335fb" candidate_uuid: "29b98c63ac7049119004395d1a2c1878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc811c7cf7a44433b7e3e601d5ec5702"
I20260923 17:19:29.856580 26444 raft_consensus.cc:3037] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:29.856580 26304 raft_consensus.cc:3037] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:29.857724 26304 raft_consensus.cc:2454] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 1.
I20260923 17:19:29.857733 26444 raft_consensus.cc:2454] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b98c63ac7049119004395d1a2c1878 in term 1.
I20260923 17:19:29.858083 26516 leader_election.cc:302] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:29.858281 26731 raft_consensus.cc:2781] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:29.858389 26731 raft_consensus.cc:686] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 1 LEADER]: Becoming Leader. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Running, Role: LEADER
I20260923 17:19:29.858565 26731 consensus_queue.cc:237] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.860322 26145 catalog_manager.cc:5495] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b98c63ac7049119004395d1a2c1878 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "29b98c63ac7049119004395d1a2c1878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } health_report { overall_health: HEALTHY } } }
I20260923 17:19:29.943868 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:29.940491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.23.214.131:34547)
0923 17:19:29.940762 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:29.940769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:29.940781 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:19:29.940821 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:29.940823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:29.940867 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:29.940943 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:29.941164 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:29.941704 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:29.942751 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:29.942995 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:29.943079 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:29.943300 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:29.943371 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:29.943435 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:29.943615 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 17:19:29.943664 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":115,"threads_started":1}
I20260923 17:19:29.950342 26304 raft_consensus.cc:1261] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Refusing update from remote peer 29b98c63ac7049119004395d1a2c1878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:29.950518 26444 raft_consensus.cc:1261] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Refusing update from remote peer 29b98c63ac7049119004395d1a2c1878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:29.950793 26723 consensus_queue.cc:1035] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:29.950950 26718 consensus_queue.cc:1035] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:29.979671 26145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:29.979825 26145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:29.981724 26145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Sending DeleteTablet for 3 replicas of tablet 0af3720844db48beb5a65a84f23335fb
I20260923 17:19:29.982424 26414 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3720844db48beb5a65a84f23335fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:29 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:46700
I20260923 17:19:29.982415 26563 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3720844db48beb5a65a84f23335fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:29 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:55640
I20260923 17:19:29.982420 26284 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af3720844db48beb5a65a84f23335fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:29 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:49824
I20260923 17:19:29.982610 26737 tablet_replica.cc:330] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: stopping tablet replica
I20260923 17:19:29.982733 26737 raft_consensus.cc:2229] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:29.982888 26738 tablet_replica.cc:330] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: stopping tablet replica
I20260923 17:19:29.982941 26737 raft_consensus.cc:2258] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:29.983029 26738 raft_consensus.cc:2229] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:29.983150 26738 raft_consensus.cc:2258] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:29.983340 26736 tablet_replica.cc:330] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: stopping tablet replica
I20260923 17:19:29.983467 26736 raft_consensus.cc:2229] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:29.983601 26736 raft_consensus.cc:2258] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:29.983603 26737 ts_tablet_manager.cc:1905] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:29.984288 26736 ts_tablet_manager.cc:1905] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:29.985980 26737 ts_tablet_manager.cc:1918] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:29.986047 26737 log.cc:1196] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/0af3720844db48beb5a65a84f23335fb
I20260923 17:19:29.986308 26737 ts_tablet_manager.cc:1939] T 0af3720844db48beb5a65a84f23335fb P 29b98c63ac7049119004395d1a2c1878: Deleting consensus metadata
I20260923 17:19:29.986617 26736 ts_tablet_manager.cc:1918] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:29.986707 26736 log.cc:1196] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/0af3720844db48beb5a65a84f23335fb
I20260923 17:19:29.986989 26736 ts_tablet_manager.cc:1939] T 0af3720844db48beb5a65a84f23335fb P dc811c7cf7a44433b7e3e601d5ec5702: Deleting consensus metadata
I20260923 17:19:29.986990 26132 catalog_manager.cc:4841] TS 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547): tablet 0af3720844db48beb5a65a84f23335fb (table test-table [id=35fc99ddfb51485e933534f3a41b26ea]) successfully deleted
I20260923 17:19:29.987026 26145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:29.987515 26145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:29.987602 26132 catalog_manager.cc:4841] TS dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057): tablet 0af3720844db48beb5a65a84f23335fb (table test-table [id=35fc99ddfb51485e933534f3a41b26ea]) successfully deleted
I20260923 17:19:29.988663 26738 ts_tablet_manager.cc:1905] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:29.991349 26738 ts_tablet_manager.cc:1918] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:29.991428 26738 log.cc:1196] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/0af3720844db48beb5a65a84f23335fb
I20260923 17:19:29.991701 26738 ts_tablet_manager.cc:1939] T 0af3720844db48beb5a65a84f23335fb P 3932029473294f2da8133390f1a8dbee: Deleting consensus metadata
I20260923 17:19:29.992125 26284 tablet_service.cc:1467] Processing CreateTablet for tablet 805cb3ae17fe4044978e3de28de1fccc (DEFAULT_TABLE table=test-table [id=1b0b0882390e46999ed0f4edfaebe8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:29.992184 26414 tablet_service.cc:1467] Processing CreateTablet for tablet 805cb3ae17fe4044978e3de28de1fccc (DEFAULT_TABLE table=test-table [id=1b0b0882390e46999ed0f4edfaebe8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:29.992385 26133 catalog_manager.cc:4841] TS 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029): tablet 0af3720844db48beb5a65a84f23335fb (table test-table [id=35fc99ddfb51485e933534f3a41b26ea]) successfully deleted
I20260923 17:19:29.992419 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805cb3ae17fe4044978e3de28de1fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:29.992431 26414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805cb3ae17fe4044978e3de28de1fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:29.992383 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 805cb3ae17fe4044978e3de28de1fccc (DEFAULT_TABLE table=test-table [id=1b0b0882390e46999ed0f4edfaebe8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:29.992612 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805cb3ae17fe4044978e3de28de1fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:29.994252 26749 tablet_bootstrap.cc:492] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702: Bootstrap starting.
I20260923 17:19:29.994637 26748 tablet_bootstrap.cc:492] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: Bootstrap starting.
I20260923 17:19:29.995002 26747 tablet_bootstrap.cc:492] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: Bootstrap starting.
I20260923 17:19:29.995182 26749 tablet_bootstrap.cc:654] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:29.995534 26748 tablet_bootstrap.cc:654] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:29.995826 26747 tablet_bootstrap.cc:654] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:29.996352 26749 tablet_bootstrap.cc:492] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702: No bootstrap required, opened a new log
I20260923 17:19:29.996433 26749 ts_tablet_manager.cc:1397] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 17:19:29.996757 26748 tablet_bootstrap.cc:492] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: No bootstrap required, opened a new log
I20260923 17:19:29.996824 26748 ts_tablet_manager.cc:1397] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:29.996775 26749 raft_consensus.cc:348] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.996946 26749 raft_consensus.cc:374] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:29.996989 26749 raft_consensus.cc:729] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc811c7cf7a44433b7e3e601d5ec5702, State: Initialized, Role: FOLLOWER
I20260923 17:19:29.997124 26749 consensus_queue.cc:260] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.997309 26748 raft_consensus.cc:348] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.997408 26748 raft_consensus.cc:374] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:29.997412 26749 ts_tablet_manager.cc:1428] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 17:19:29.997440 26748 raft_consensus.cc:729] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3932029473294f2da8133390f1a8dbee, State: Initialized, Role: FOLLOWER
I20260923 17:19:29.997561 26748 consensus_queue.cc:260] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.997840 26748 ts_tablet_manager.cc:1428] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:29.998584 26747 tablet_bootstrap.cc:492] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: No bootstrap required, opened a new log
I20260923 17:19:29.998682 26747 ts_tablet_manager.cc:1397] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 17:19:29.999125 26747 raft_consensus.cc:348] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.999249 26747 raft_consensus.cc:374] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:29.999291 26747 raft_consensus.cc:729] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b98c63ac7049119004395d1a2c1878, State: Initialized, Role: FOLLOWER
I20260923 17:19:29.999436 26747 consensus_queue.cc:260] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:29.999691 26747 ts_tablet_manager.cc:1428] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:19:30.115774 26754 raft_consensus.cc:482] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:30.115953 26754 raft_consensus.cc:504] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:30.116466 26754 leader_election.cc:288] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029), 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547)
I20260923 17:19:30.116971 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cb3ae17fe4044978e3de28de1fccc" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b98c63ac7049119004395d1a2c1878" is_pre_election: true
I20260923 17:19:30.116972 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cb3ae17fe4044978e3de28de1fccc" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee" is_pre_election: true
I20260923 17:19:30.117100 26568 raft_consensus.cc:2454] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 0.
I20260923 17:19:30.117151 26444 raft_consensus.cc:2454] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 0.
I20260923 17:19:30.117478 26236 leader_election.cc:302] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:30.117652 26754 raft_consensus.cc:2781] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:30.117730 26754 raft_consensus.cc:482] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:30.117771 26754 raft_consensus.cc:3037] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:30.118872 26754 raft_consensus.cc:504] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:30.119293 26754 leader_election.cc:288] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 election: Requested vote from peers 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029), 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547)
I20260923 17:19:30.119696 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cb3ae17fe4044978e3de28de1fccc" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b98c63ac7049119004395d1a2c1878"
I20260923 17:19:30.119762 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805cb3ae17fe4044978e3de28de1fccc" candidate_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3932029473294f2da8133390f1a8dbee"
I20260923 17:19:30.119869 26444 raft_consensus.cc:3037] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:30.119870 26568 raft_consensus.cc:3037] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:30.121008 26444 raft_consensus.cc:2454] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 1.
I20260923 17:19:30.121008 26568 raft_consensus.cc:2454] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc811c7cf7a44433b7e3e601d5ec5702 in term 1.
I20260923 17:19:30.121331 26236 leader_election.cc:302] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b98c63ac7049119004395d1a2c1878, dc811c7cf7a44433b7e3e601d5ec5702; no voters:
I20260923 17:19:30.121521 26754 raft_consensus.cc:2781] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:30.121716 26754 raft_consensus.cc:686] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [term 1 LEADER]: Becoming Leader. State: Replica: dc811c7cf7a44433b7e3e601d5ec5702, State: Running, Role: LEADER
I20260923 17:19:30.121881 26754 consensus_queue.cc:237] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } }
I20260923 17:19:30.123680 26147 catalog_manager.cc:5495] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129). New cstate: current_term: 1 leader_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc811c7cf7a44433b7e3e601d5ec5702" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20260923 17:19:30.164005 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:30.160876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.23.214.129:37057)
0923 17:19:30.161147 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:30.161154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:30.161165 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:19:30.161205 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:30.161207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:30.161248 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:30.161346 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:30.161567 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:30.162174 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:30.163053 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:30.163278 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:30.163384 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:30.163581 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:30.163650 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:30.163708 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:30.163839 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 17:19:30.163884 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":128,"threads_started":1}
I20260923 17:19:30.171016 26568 raft_consensus.cc:1261] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Refusing update from remote peer dc811c7cf7a44433b7e3e601d5ec5702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:30.171079 26444 raft_consensus.cc:1261] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Refusing update from remote peer dc811c7cf7a44433b7e3e601d5ec5702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:30.171468 26764 consensus_queue.cc:1035] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b98c63ac7049119004395d1a2c1878" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:30.171633 26754 consensus_queue.cc:1035] T 805cb3ae17fe4044978e3de28de1fccc P dc811c7cf7a44433b7e3e601d5ec5702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3932029473294f2da8133390f1a8dbee" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:19:30.186586 26720 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:19:31.745303 26145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:31.745460 26145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:31.747987 26145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 849052ab5b8842b2993f682b85626d56: Sending DeleteTablet for 3 replicas of tablet 805cb3ae17fe4044978e3de28de1fccc
I20260923 17:19:31.748569 26284 tablet_service.cc:1514] Processing DeleteTablet for tablet 805cb3ae17fe4044978e3de28de1fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:31 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:49824
I20260923 17:19:31.748595 26414 tablet_service.cc:1514] Processing DeleteTablet for tablet 805cb3ae17fe4044978e3de28de1fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:31 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:46700
I20260923 17:19:31.748842 26563 tablet_service.cc:1514] Processing DeleteTablet for tablet 805cb3ae17fe4044978e3de28de1fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:31 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:55640
I20260923 17:19:31.748976 26794 tablet_replica.cc:330] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: stopping tablet replica
I20260923 17:19:31.749119 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 26219
I20260923 17:19:31.750001 26794 raft_consensus.cc:2229] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:31.750165 26794 raft_consensus.cc:2258] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:31.750442 26795 tablet_replica.cc:330] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: stopping tablet replica
I20260923 17:19:31.750622 26795 raft_consensus.cc:2229] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:31.750775 26795 raft_consensus.cc:2258] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:31.751057 26794 ts_tablet_manager.cc:1905] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:31.751503 26795 ts_tablet_manager.cc:1905] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:31.753775 26795 ts_tablet_manager.cc:1918] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:19:31.753872 26795 log.cc:1196] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/805cb3ae17fe4044978e3de28de1fccc
I20260923 17:19:31.754187 26795 ts_tablet_manager.cc:1939] T 805cb3ae17fe4044978e3de28de1fccc P 29b98c63ac7049119004395d1a2c1878: Deleting consensus metadata
I20260923 17:19:31.754591 26794 ts_tablet_manager.cc:1918] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:19:31.754676 26794 log.cc:1196] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/805cb3ae17fe4044978e3de28de1fccc
I20260923 17:19:31.755014 26794 ts_tablet_manager.cc:1939] T 805cb3ae17fe4044978e3de28de1fccc P 3932029473294f2da8133390f1a8dbee: Deleting consensus metadata
I20260923 17:19:31.755170 26132 catalog_manager.cc:4841] TS 29b98c63ac7049119004395d1a2c1878 (127.23.214.131:34547): tablet 805cb3ae17fe4044978e3de28de1fccc (table test-table [id=1b0b0882390e46999ed0f4edfaebe8ca]) successfully deleted
I20260923 17:19:31.755859 26133 catalog_manager.cc:4841] TS 3932029473294f2da8133390f1a8dbee (127.23.214.130:43029): tablet 805cb3ae17fe4044978e3de28de1fccc (table test-table [id=1b0b0882390e46999ed0f4edfaebe8ca]) successfully deleted
W20260923 17:19:31.758311 26132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:37057 (error 108)
I20260923 17:19:31.758546 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 26359
I20260923 17:19:31.760737 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:31.760210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37057 (local address 127.0.0.1:33070)
0923 17:19:31.760527 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:31.760629 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
W20260923 17:19:31.761011 26132 catalog_manager.cc:4584] TS dc811c7cf7a44433b7e3e601d5ec5702 (127.23.214.129:37057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 805cb3ae17fe4044978e3de28de1fccc: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37057: connect: Connection refused (error 111)
I20260923 17:19:31.765156 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 26499
I20260923 17:19:31.772246 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 26115
2026-09-23T17:19:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7062 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:31 dist-test-slave-9q7f krb5kdc[26800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:31 dist-test-slave-9q7f krb5kdc[26800](info): set up 2 sockets
Sep 23 17:19:31 dist-test-slave-9q7f krb5kdc[26800](info): commencing operation
krb5kdc: starting...
W20260923 17:19:32.824090 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:32 dist-test-slave-9q7f krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:32.949791 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35639
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:33.064965 26813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:33.065295 26813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:33.065402 26813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:33.069566 26813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:33.069669 26813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:33.069706 26813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:33.069744 26813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:33.069783 26813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:33.075429 26813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:33.076567 26813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:33.077646 26813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:33.086582 26821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:33.086572 26819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:33.086771 26813 server_base.cc:1034] running on GCE node
W20260923 17:19:33.086575 26818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:33.087519 26813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:33.088675 26813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:33.089955 26813 hybrid_clock.cc:648] HybridClock initialized: now 1790183973089927 us; error 29 us; skew 500 ppm
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:33.092770 26813 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:33.093709 26813 webserver.cc:460] Webserver started at http://127.23.214.190:39023/ using document root <none> and password file <none>
I20260923 17:19:33.094336 26813 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:33.094398 26813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:33.094612 26813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:33.096679 26813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "bacf60bcb14b79508cd713d9f49aa182"
server_key_iv: "db3fc538a7b0b2eaf014741bb273da37"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.097251 26813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "bacf60bcb14b79508cd713d9f49aa182"
server_key_iv: "db3fc538a7b0b2eaf014741bb273da37"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.101039 26813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:19:33.103629 26828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.104933 26813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:19:33.105072 26813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "bacf60bcb14b79508cd713d9f49aa182"
server_key_iv: "db3fc538a7b0b2eaf014741bb273da37"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.105190 26813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:33.141100 26813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:33.144888 26813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:33.145139 26813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:33.152688 26813 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:35639
I20260923 17:19:33.152733 26880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:35639 every 8 connection(s)
I20260923 17:19:33.153748 26813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:33.156939 26881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:33.157387 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 26813
I20260923 17:19:33.157480 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:33.157806 24410 external_mini_cluster.cc:1428] Setting key 90e54a969b61537aa6fd39f3deb08ba8
I20260923 17:19:33.164207 26881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f: Bootstrap starting.
I20260923 17:19:33.166934 26881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:33.167826 26881 log.cc:824] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:33.170372 26881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f: No bootstrap required, opened a new log
I20260923 17:19:33.173955 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:33.159383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.23.214.190:35639)
0923 17:19:33.159867 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:33.159888 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:33.159948 (+ 60us) server_negotiation.cc:407] Connection header received
0923 17:19:33.160797 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:33.160822 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:33.161281 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:33.161663 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:33.162507 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.164000 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.165019 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.165751 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.165960 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:33.168893 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:33.168899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:33.168914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:33.168950 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:33.170863 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.171379 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.171385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.171390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.171510 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.171821 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.171824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.171828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.172250 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:33.172461 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:33.172758 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 17:19:33.173083 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":140,"threads_started":1}
I20260923 17:19:33.174190 26881 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35639 } }
I20260923 17:19:33.174496 26881 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:33.174561 26881 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c05b361ba8462b8e9eda8d1d2dd95f, State: Initialized, Role: FOLLOWER
I20260923 17:19:33.175038 26881 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35639 } }
I20260923 17:19:33.175248 26881 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:33.175330 26881 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:33.175455 26881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:33.176640 26881 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35639 } }
I20260923 17:19:33.176998 26881 leader_election.cc:302] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1c05b361ba8462b8e9eda8d1d2dd95f; no voters:
I20260923 17:19:33.177484 26881 leader_election.cc:288] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:33.177577 26886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:33.177824 26886 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [term 1 LEADER]: Becoming Leader. State: Replica: a1c05b361ba8462b8e9eda8d1d2dd95f, State: Running, Role: LEADER
I20260923 17:19:33.178171 26886 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35639 } }
I20260923 17:19:33.178859 26881 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:33.180020 26887 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35639 } } }
I20260923 17:19:33.180054 26888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1c05b361ba8462b8e9eda8d1d2dd95f. Latest consensus state: current_term: 1 leader_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c05b361ba8462b8e9eda8d1d2dd95f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35639 } } }
I20260923 17:19:33.180179 26888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:33.180179 26887 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:33.180630 26895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:33.184026 26895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:33.190335 26895 catalog_manager.cc:1347] Generated new cluster ID: dcd47e2d7f6c4b9fa94278a34c170470
I20260923 17:19:33.190456 26895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:33.204128 26895 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:33.205170 26895 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:33.214972 26895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1c05b361ba8462b8e9eda8d1d2dd95f: Generated new TSK 0
I20260923 17:19:33.215888 26895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:33.249195 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35639
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:33.389374 26907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:33.389683 26907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:33.389734 26907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:33.394085 26907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:33.394160 26907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:33.394265 26907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:33.400029 26907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:33.401201 26907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:33.402329 26907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:33.412343 26913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:33.412343 26912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:33.412390 26915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:33.412755 26907 server_base.cc:1034] running on GCE node
I20260923 17:19:33.413230 26907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:33.413976 26907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:33.415207 26907 hybrid_clock.cc:648] HybridClock initialized: now 1790183973415184 us; error 36 us; skew 500 ppm
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:33.418282 26907 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:33.419275 26907 webserver.cc:460] Webserver started at http://127.23.214.129:46599/ using document root <none> and password file <none>
I20260923 17:19:33.419875 26907 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:33.419938 26907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:33.420159 26907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:33.422434 26907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "3d162e30c42941e1951410d05e2a30ca"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "ccf507022258efc46e904aa46b27667f"
server_key_iv: "2266426332785d66b4958349972ab3c0"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.423008 26907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "3d162e30c42941e1951410d05e2a30ca"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "ccf507022258efc46e904aa46b27667f"
server_key_iv: "2266426332785d66b4958349972ab3c0"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.426929 26907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:33.429558 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.431022 26907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:19:33.431123 26907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "3d162e30c42941e1951410d05e2a30ca"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "ccf507022258efc46e904aa46b27667f"
server_key_iv: "2266426332785d66b4958349972ab3c0"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.431223 26907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:33.446663 26907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:33.450239 26907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:33.450415 26907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:33.450992 26907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:33.451854 26907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:33.451903 26907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.451949 26907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:33.451965 26907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.460744 26907 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:34577
I20260923 17:19:33.460786 27035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:34577 every 8 connection(s)
I20260923 17:19:33.461659 26907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:33.466285 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 26907
I20260923 17:19:33.466431 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:33.466699 24410 external_mini_cluster.cc:1428] Setting key e6df2d280872c5ee44ba608e410d4c55
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:19:33.476900 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:33.463682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:44825 (local address 127.23.214.190:35639)
0923 17:19:33.463876 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:33.463881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:33.464975 (+ 1094us) server_negotiation.cc:407] Connection header received
0923 17:19:33.465811 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:33.465813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:33.465861 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:33.465997 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:33.467345 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.468109 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.469923 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.470351 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.470473 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:33.473191 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:33.473202 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:33.473204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:33.473229 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:33.474801 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.475353 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.475357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.475359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.475440 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.475787 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.475792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.475794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.475986 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:33.476085 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:33.476600 (+ 515us) server_negotiation.cc:299] Negotiation successful
0923 17:19:33.476732 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:19:33.477919 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:33.464123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35639 (local address 127.23.214.129:44825)
0923 17:19:33.464811 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:33.464853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:33.465630 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:33.466169 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:33.466177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:33.466621 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:33.467171 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:33.467206 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.468253 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.468258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:33.469767 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:33.469782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.470505 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.470508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:33.470651 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:33.471280 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:33.471296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:33.473013 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:33.474950 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:33.474955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:33.474972 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:33.475238 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:33.475540 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:33.475544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:33.475546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:33.475691 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:33.476099 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:33.476108 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:33.476365 (+ 257us) client_negotiation.cc:769] Sending connection context
0923 17:19:33.476561 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 17:19:33.476913 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:33.479278 27036 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35639
I20260923 17:19:33.479626 27036 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:33.480170 27036 heartbeater.cc:511] Master 127.23.214.190:35639 requested a full tablet report, sending...
I20260923 17:19:33.482077 26845 ts_manager.cc:194] Registered new tserver with Master: 3d162e30c42941e1951410d05e2a30ca (127.23.214.129:34577)
I20260923 17:19:33.483260 26845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:44825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:33.496201 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35639
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:33.612795 27041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:33.613118 27041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:33.613170 27041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:33.617643 27041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:33.617715 27041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:33.617779 27041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:33.623416 27041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:33.624527 27041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:33.625552 27041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:33.635565 27047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:33.635565 27049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:33.635634 27046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:33.635818 27041 server_base.cc:1034] running on GCE node
I20260923 17:19:33.636370 27041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:33.637073 27041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:33.638237 27041 hybrid_clock.cc:648] HybridClock initialized: now 1790183973638218 us; error 35 us; skew 500 ppm
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:33.641064 27041 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:33.642135 27041 webserver.cc:460] Webserver started at http://127.23.214.130:38565/ using document root <none> and password file <none>
I20260923 17:19:33.642720 27041 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:33.642786 27041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:33.642956 27041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:33.645133 27041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "1194d03dd2304186a4c0d34b02c5e777"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "e0bf9ccc58e0cbe31d7bb612763193a4"
server_key_iv: "3b5dfb2da32acf2977b7d76dcbd923cb"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.645711 27041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "1194d03dd2304186a4c0d34b02c5e777"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "e0bf9ccc58e0cbe31d7bb612763193a4"
server_key_iv: "3b5dfb2da32acf2977b7d76dcbd923cb"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.650126 27041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:33.652782 27056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.654138 27041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:33.654307 27041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "1194d03dd2304186a4c0d34b02c5e777"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "e0bf9ccc58e0cbe31d7bb612763193a4"
server_key_iv: "3b5dfb2da32acf2977b7d76dcbd923cb"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.654443 27041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:33.681257 27041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:33.684744 27041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:33.684966 27041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:33.685616 27041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:33.686542 27041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:33.686614 27041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.686687 27041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:33.686739 27041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.696257 27041 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:45829
I20260923 17:19:33.696275 27169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:45829 every 8 connection(s)
I20260923 17:19:33.697232 27041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:33.703258 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27041
I20260923 17:19:33.703356 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:33.703624 24410 external_mini_cluster.cc:1428] Setting key ca95b6e672cae1c937519c385c1bb98e
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:33.712443 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:33.699181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:51597 (local address 127.23.214.190:35639)
0923 17:19:33.699369 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:33.699372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:33.700371 (+ 999us) server_negotiation.cc:407] Connection header received
0923 17:19:33.701343 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:33.701348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:33.701407 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:33.701533 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:33.702880 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.703485 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.704818 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.705135 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.705260 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:33.708254 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:33.708257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:33.708259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:33.708283 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:33.710091 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.710758 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.710760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.710761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.710834 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.711279 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.711281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.711282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.711459 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:33.711532 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:33.712205 (+ 673us) server_negotiation.cc:299] Negotiation successful
0923 17:19:33.712310 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 17:19:33.713748 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:33.699517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35639 (local address 127.23.214.130:51597)
0923 17:19:33.700197 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:33.700251 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:33.701102 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:33.701724 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:33.701731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:33.702205 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:33.702709 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:33.702743 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.703611 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.703614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:33.704678 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:33.704688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.705300 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.705303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:33.705484 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:33.706152 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:33.706169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:33.708052 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:33.710209 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:33.710228 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:33.710250 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:33.710653 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:33.710943 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:33.710950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:33.710954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:33.711165 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:33.711563 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:33.711573 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:33.711933 (+ 360us) client_negotiation.cc:769] Sending connection context
0923 17:19:33.712162 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 17:19:33.712566 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:19:33.715022 27170 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35639
I20260923 17:19:33.715319 27170 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:33.715839 27170 heartbeater.cc:511] Master 127.23.214.190:35639 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
I20260923 17:19:33.717013 26845 ts_manager.cc:194] Registered new tserver with Master: 1194d03dd2304186a4c0d34b02c5e777 (127.23.214.130:45829)
I20260923 17:19:33.717860 26845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:51597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:33.733985 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35639
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:33.849584 27175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:33.849900 27175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:33.849964 27175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:33.854002 27175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:33.854070 27175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:33.854130 27175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:33.859644 27175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:33.860810 27175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:33.861953 27175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:33.871887 27180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:33.871896 27181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:33.871914 27183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:33.872344 27175 server_base.cc:1034] running on GCE node
I20260923 17:19:33.872745 27175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:33.873378 27175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:33.874562 27175 hybrid_clock.cc:648] HybridClock initialized: now 1790183973874544 us; error 26 us; skew 500 ppm
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:33.877486 27175 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:33.878492 27175 webserver.cc:460] Webserver started at http://127.23.214.131:43233/ using document root <none> and password file <none>
I20260923 17:19:33.879040 27175 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:33.879098 27175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:33.879328 27175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:33.881605 27175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "d59806e5d4a249e0ad48cdb8e040de13"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "3ce27e2d9ed0734248b3e9cbc66591ae"
server_key_iv: "9426ba4c472918a568df172e5423b412"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.882192 27175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "d59806e5d4a249e0ad48cdb8e040de13"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "3ce27e2d9ed0734248b3e9cbc66591ae"
server_key_iv: "9426ba4c472918a568df172e5423b412"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.886246 27175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:33.888799 27190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.890244 27175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:33.890381 27175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "d59806e5d4a249e0ad48cdb8e040de13"
format_stamp: "Formatted at 2026-09-23 17:19:33 on dist-test-slave-9q7f"
server_key: "3ce27e2d9ed0734248b3e9cbc66591ae"
server_key_iv: "9426ba4c472918a568df172e5423b412"
server_key_version: "encryptionkey@0"
I20260923 17:19:33.890501 27175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:33.909585 27175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:33.913045 27175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:33.913295 27175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:33.914004 27175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:33.914903 27175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:33.914976 27175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.915048 27175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:33.915092 27175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:33.924482 27175 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:46639
I20260923 17:19:33.924499 27303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:46639 every 8 connection(s)
I20260923 17:19:33.925470 27175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:33.931160 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27175
I20260923 17:19:33.931290 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:33.931550 24410 external_mini_cluster.cc:1428] Setting key 16c85407b4fa59686299c3e1ec4fbb84
Sep 23 17:19:33 dist-test-slave-9q7f krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183973, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:33.940035 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:33.927397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:40557 (local address 127.23.214.190:35639)
0923 17:19:33.927559 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:33.927563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:33.928618 (+ 1055us) server_negotiation.cc:407] Connection header received
0923 17:19:33.929595 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:33.929598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:33.929658 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:33.929773 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:33.931065 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.931742 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.933233 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.933483 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.933597 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:33.936415 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:33.936418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:33.936421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:33.936456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:33.938044 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.938585 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.938588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.938589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.938665 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:33.939025 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:33.939027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:33.939028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:33.939209 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:33.939286 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:33.939793 (+ 507us) server_negotiation.cc:299] Negotiation successful
0923 17:19:33.939908 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:19:33.940935 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:33.927803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35639 (local address 127.23.214.131:40557)
0923 17:19:33.928460 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:33.928504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:33.929354 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:33.929996 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:33.930003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:33.930375 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:33.930891 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:33.930913 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.931875 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.931881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:33.933114 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:33.933128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:33.933635 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:33.933639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:33.933816 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:33.934632 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:33.934649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:33.936252 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:33.938158 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:33.938164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:33.938179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:33.938485 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:33.938780 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:33.938783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:33.938784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:33.938940 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:33.939305 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:33.939311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:33.939567 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 17:19:33.939753 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 17:19:33.940027 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":109,"threads_started":1}
I20260923 17:19:33.942229 27304 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35639
I20260923 17:19:33.942451 27304 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:33.943001 27304 heartbeater.cc:511] Master 127.23.214.190:35639 requested a full tablet report, sending...
I20260923 17:19:33.944133 26845 ts_manager.cc:194] Registered new tserver with Master: d59806e5d4a249e0ad48cdb8e040de13 (127.23.214.131:46639)
I20260923 17:19:33.944873 26845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:40557
I20260923 17:19:33.945979 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:19:33.959800 26884 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:33.957793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.23.214.190:35639)
0923 17:19:33.957990 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:33.957995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:33.958305 (+ 310us) server_negotiation.cc:407] Connection header received
0923 17:19:33.958444 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:33.958446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:33.958492 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:33.958566 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:33.959649 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47234: BlockingRecv error: recv got EOF from 127.0.0.1:47234 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 17:19:33.960760 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 26907
I20260923 17:19:33.967221 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27041
I20260923 17:19:33.973173 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27175
I20260923 17:19:33.978971 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 26813
2026-09-23T17:19:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2205 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:34 dist-test-slave-9q7f krb5kdc[27316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:34 dist-test-slave-9q7f krb5kdc[27316](info): set up 2 sockets
Sep 23 17:19:34 dist-test-slave-9q7f krb5kdc[27316](info): commencing operation
krb5kdc: starting...
W20260923 17:19:35.055935 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:35 dist-test-slave-9q7f krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:35.180824 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45565
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:35.292770 27329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:35.293063 27329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:35.293107 27329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:35.297224 27329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:35.297293 27329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:35.297313 27329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:35.297335 27329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:35.297351 27329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:35.302848 27329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:35.303905 27329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:35.304883 27329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:35.313675 27337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:35.313657 27335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:35.313684 27334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:35.313717 27329 server_base.cc:1034] running on GCE node
I20260923 17:19:35.314597 27329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:35.315805 27329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:35.317000 27329 hybrid_clock.cc:648] HybridClock initialized: now 1790183975316966 us; error 58 us; skew 500 ppm
Sep 23 17:19:35 dist-test-slave-9q7f krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183975, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:35.326215 27329 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:35.327399 27329 webserver.cc:460] Webserver started at http://127.23.214.190:39361/ using document root <none> and password file <none>
I20260923 17:19:35.328008 27329 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:35.328101 27329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:35.328351 27329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:35.330576 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "1d51e552183b4fbdaa4697e38e152041"
format_stamp: "Formatted at 2026-09-23 17:19:35 on dist-test-slave-9q7f"
server_key: "2d29a1b9ae3c585c1c78ab1803444342"
server_key_iv: "f2b7b4cc4ff69fe0df8e560b8cd5d220"
server_key_version: "encryptionkey@0"
I20260923 17:19:35.331169 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "1d51e552183b4fbdaa4697e38e152041"
format_stamp: "Formatted at 2026-09-23 17:19:35 on dist-test-slave-9q7f"
server_key: "2d29a1b9ae3c585c1c78ab1803444342"
server_key_iv: "f2b7b4cc4ff69fe0df8e560b8cd5d220"
server_key_version: "encryptionkey@0"
I20260923 17:19:35.335009 27329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:19:35.337510 27344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:35.338830 27329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:19:35.339277 27329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "1d51e552183b4fbdaa4697e38e152041"
format_stamp: "Formatted at 2026-09-23 17:19:35 on dist-test-slave-9q7f"
server_key: "2d29a1b9ae3c585c1c78ab1803444342"
server_key_iv: "f2b7b4cc4ff69fe0df8e560b8cd5d220"
server_key_version: "encryptionkey@0"
I20260923 17:19:35.339422 27329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:35.355229 27329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:35.359189 27329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:35.359436 27329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:35.367650 27329 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:45565
I20260923 17:19:35.367956 27396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:45565 every 8 connection(s)
I20260923 17:19:35.368873 27329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:35.372215 27397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:35.377686 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27329
I20260923 17:19:35.377847 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:35.378194 24410 external_mini_cluster.cc:1428] Setting key 07038b938416727636528132296e6968
I20260923 17:19:35.378396 27397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041: Bootstrap starting.
I20260923 17:19:35.381014 27397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:35.382010 27397 log.cc:824] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:35.384377 27397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041: No bootstrap required, opened a new log
Sep 23 17:19:35 dist-test-slave-9q7f krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:35.389040 27397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51e552183b4fbdaa4697e38e152041" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45565 } }
I20260923 17:19:35.389338 27397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:35.389427 27397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d51e552183b4fbdaa4697e38e152041, State: Initialized, Role: FOLLOWER
I20260923 17:19:35.390065 27397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51e552183b4fbdaa4697e38e152041" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45565 } }
I20260923 17:19:35.390247 27397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:35.390343 27397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:35.390448 27397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:35.391409 27397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51e552183b4fbdaa4697e38e152041" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45565 } }
I20260923 17:19:35.391847 27397 leader_election.cc:302] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d51e552183b4fbdaa4697e38e152041; no voters:
I20260923 17:19:35.392228 27397 leader_election.cc:288] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:35.392313 27402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:35.392549 27402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [term 1 LEADER]: Becoming Leader. State: Replica: 1d51e552183b4fbdaa4697e38e152041, State: Running, Role: LEADER
I20260923 17:19:35.393092 27402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51e552183b4fbdaa4697e38e152041" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45565 } }
I20260923 17:19:35.393729 27397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:35.394572 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:35.379640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.23.214.190:45565)
0923 17:19:35.380100 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:35.380117 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:35.380153 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:19:35.380845 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:35.380864 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:35.381285 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:35.381597 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:35.382685 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:35.383801 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:35.385011 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:35.385522 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:35.385658 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:35.388542 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:35.388548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:35.388564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:35.388595 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:35.390760 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:35.391259 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:35.391264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:35.391270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:35.391403 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:35.391733 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:35.391736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:35.391738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:35.392236 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:35.392444 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:35.393159 (+ 715us) server_negotiation.cc:299] Negotiation successful
0923 17:19:35.393477 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260923 17:19:35.394910 27403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d51e552183b4fbdaa4697e38e152041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51e552183b4fbdaa4697e38e152041" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45565 } } }
I20260923 17:19:35.395044 27403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:35.395382 27404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d51e552183b4fbdaa4697e38e152041. Latest consensus state: current_term: 1 leader_uuid: "1d51e552183b4fbdaa4697e38e152041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51e552183b4fbdaa4697e38e152041" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45565 } } }
I20260923 17:19:35.395481 27404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:35.397964 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 17:19:35.399098 27417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:19:35.399186 27417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:19:35.399253 27411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:35.400288 27411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:35.406170 27411 catalog_manager.cc:1347] Generated new cluster ID: 5746029a1c914309abff9350e5229aa2
I20260923 17:19:35.406275 27411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:35.418838 27411 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:35.419989 27411 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:35.424077 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:35.418265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.23.214.190:45565)
0923 17:19:35.418395 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:35.418400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:35.418664 (+ 264us) server_negotiation.cc:407] Connection header received
0923 17:19:35.418908 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:35.418911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:35.418970 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:35.419059 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:35.419704 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:35.420379 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:35.421573 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:35.421958 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:35.422057 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:35.423013 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:35.423017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:35.423020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:35.423588 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:35.423619 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:35.423637 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:35.423719 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:35.423852 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 17:19:35.423931 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:19:35.429692 27411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d51e552183b4fbdaa4697e38e152041: Generated new TSK 0
I20260923 17:19:35.430497 27411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:19:35.439311 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27329
2026-09-23T17:19:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1457 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:35 dist-test-slave-9q7f krb5kdc[27430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:35 dist-test-slave-9q7f krb5kdc[27430](info): set up 2 sockets
Sep 23 17:19:35 dist-test-slave-9q7f krb5kdc[27430](info): commencing operation
krb5kdc: starting...
W20260923 17:19:36.480151 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:36 dist-test-slave-9q7f krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:36.613306 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45879
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:45879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:36.727828 27443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:36.728210 27443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:36.728304 27443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:36.732592 27443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:36.732717 27443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:36.732767 27443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:36.732807 27443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:36.732841 27443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:36.738534 27443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:45879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:36.739809 27443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:36.740975 27443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:36.750312 27449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:36.750356 27451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:36.750605 27443 server_base.cc:1034] running on GCE node
W20260923 17:19:36.750622 27448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:36.751343 27443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:36.752620 27443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:36.753849 27443 hybrid_clock.cc:648] HybridClock initialized: now 1790183976753818 us; error 42 us; skew 500 ppm
Sep 23 17:19:36 dist-test-slave-9q7f krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183976, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:36.757109 27443 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:36.758315 27443 webserver.cc:460] Webserver started at http://127.23.214.190:35289/ using document root <none> and password file <none>
I20260923 17:19:36.758939 27443 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:36.759047 27443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:36.759308 27443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:36.761494 27443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "4259fba3adab4e75886daf45c45ab67e"
format_stamp: "Formatted at 2026-09-23 17:19:36 on dist-test-slave-9q7f"
server_key: "8a3b8988332c87ed214c17b694fccafe"
server_key_iv: "72b22ac00d692449c1c28d9d1318e590"
server_key_version: "encryptionkey@0"
I20260923 17:19:36.762147 27443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "4259fba3adab4e75886daf45c45ab67e"
format_stamp: "Formatted at 2026-09-23 17:19:36 on dist-test-slave-9q7f"
server_key: "8a3b8988332c87ed214c17b694fccafe"
server_key_iv: "72b22ac00d692449c1c28d9d1318e590"
server_key_version: "encryptionkey@0"
I20260923 17:19:36.766093 27443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:36.768761 27458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:36.770269 27443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:36.770423 27443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "4259fba3adab4e75886daf45c45ab67e"
format_stamp: "Formatted at 2026-09-23 17:19:36 on dist-test-slave-9q7f"
server_key: "8a3b8988332c87ed214c17b694fccafe"
server_key_iv: "72b22ac00d692449c1c28d9d1318e590"
server_key_version: "encryptionkey@0"
I20260923 17:19:36.770551 27443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:36.796880 27443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:36.800206 27443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:36.800428 27443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:36.807840 27443 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:45879
I20260923 17:19:36.808154 27510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:45879 every 8 connection(s)
I20260923 17:19:36.808909 27443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:36.810496 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27443
I20260923 17:19:36.810623 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:36.810935 24410 external_mini_cluster.cc:1428] Setting key a011a3a21906adc70b663d9cbed6e0d4
I20260923 17:19:36.812558 27511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:36.819689 27511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e: Bootstrap starting.
Sep 23 17:19:36 dist-test-slave-9q7f krb5kdc[27430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:36.822512 27511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:36.823402 27511 log.cc:824] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:36.825433 27511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e: No bootstrap required, opened a new log
I20260923 17:19:36.827147 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:36.812437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.23.214.190:45879)
0923 17:19:36.812886 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:36.812913 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:36.812951 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:19:36.813632 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:36.813650 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:36.814082 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:36.814393 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:36.815411 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:36.816534 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:36.817735 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:36.818374 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:36.818510 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:36.821725 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:36.821731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:36.821745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:36.821778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:36.823966 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:36.824470 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:36.824476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:36.824481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:36.824596 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:36.824925 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:36.824930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:36.824931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:36.825359 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:36.825561 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:36.825869 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 17:19:36.826204 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":166,"threads_started":1}
I20260923 17:19:36.828928 27511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4259fba3adab4e75886daf45c45ab67e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45879 } }
I20260923 17:19:36.829200 27511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:36.829258 27511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4259fba3adab4e75886daf45c45ab67e, State: Initialized, Role: FOLLOWER
I20260923 17:19:36.829736 27511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4259fba3adab4e75886daf45c45ab67e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45879 } }
I20260923 17:19:36.829926 27511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:36.830076 27511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:36.830219 27511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:36.831197 27511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4259fba3adab4e75886daf45c45ab67e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45879 } }
I20260923 17:19:36.831560 27511 leader_election.cc:302] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4259fba3adab4e75886daf45c45ab67e; no voters:
I20260923 17:19:36.831910 27511 leader_election.cc:288] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:36.832051 27516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:36.832360 27516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [term 1 LEADER]: Becoming Leader. State: Replica: 4259fba3adab4e75886daf45c45ab67e, State: Running, Role: LEADER
I20260923 17:19:36.832660 27516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4259fba3adab4e75886daf45c45ab67e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45879 } }
I20260923 17:19:36.833209 27511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:36.834375 27518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4259fba3adab4e75886daf45c45ab67e. Latest consensus state: current_term: 1 leader_uuid: "4259fba3adab4e75886daf45c45ab67e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4259fba3adab4e75886daf45c45ab67e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45879 } } }
I20260923 17:19:36.834518 27518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:36.834980 27525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:36.835337 27517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4259fba3adab4e75886daf45c45ab67e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4259fba3adab4e75886daf45c45ab67e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45879 } } }
I20260923 17:19:36.835430 27517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:36.838456 27525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:36.844712 27525 catalog_manager.cc:1347] Generated new cluster ID: 3b20e4e2c32142a5bb8e153d400b73ce
I20260923 17:19:36.844810 27525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:36.855171 27525 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:36.856223 27525 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:36.870280 27525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4259fba3adab4e75886daf45c45ab67e: Generated new TSK 0
I20260923 17:19:36.871129 27525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:36.901841 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45879
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:37.023346 27537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:37.023669 27537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:37.023715 27537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:37.027935 27537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:37.028014 27537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:37.028146 27537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:37.036966 27537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:37.038172 27537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:37.039254 27537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:37.049221 27542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:37.049221 27545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:37.049340 27543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:37.049474 27537 server_base.cc:1034] running on GCE node
I20260923 17:19:37.050146 27537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:37.050916 27537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:37.052093 27537 hybrid_clock.cc:648] HybridClock initialized: now 1790183977052037 us; error 71 us; skew 500 ppm
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:37.055172 27537 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:37.056183 27537 webserver.cc:460] Webserver started at http://127.23.214.129:38495/ using document root <none> and password file <none>
I20260923 17:19:37.056797 27537 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:37.056891 27537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:37.057133 27537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:37.059420 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "a365d265fe31496790c13b4fa91be659"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "d9982da07a8063647be8879ee5039166"
server_key_iv: "538cee9ead66e6e43f76da4cc2c35aa4"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.060041 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "a365d265fe31496790c13b4fa91be659"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "d9982da07a8063647be8879ee5039166"
server_key_iv: "538cee9ead66e6e43f76da4cc2c35aa4"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.064077 27537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:37.066793 27552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.068222 27537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:19:37.068392 27537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "a365d265fe31496790c13b4fa91be659"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "d9982da07a8063647be8879ee5039166"
server_key_iv: "538cee9ead66e6e43f76da4cc2c35aa4"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.068526 27537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:37.097720 27537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:37.101642 27537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:37.101949 27537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:37.102627 27537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:37.103520 27537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:37.103596 27537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.103673 27537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:37.103717 27537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.113296 27537 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:42615
I20260923 17:19:37.113318 27665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:42615 every 8 connection(s)
I20260923 17:19:37.114332 27537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:37.119903 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27537
I20260923 17:19:37.120018 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:37.120339 24410 external_mini_cluster.cc:1428] Setting key f3b2078a50aa494e51c2adb4cf29bb4c
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:19:37.130050 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:37.116445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:53973 (local address 127.23.214.190:45879)
0923 17:19:37.116610 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:37.116615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:37.117630 (+ 1015us) server_negotiation.cc:407] Connection header received
0923 17:19:37.119110 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:37.119113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:37.119165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:37.119240 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:37.120719 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.121337 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.122738 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.122996 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.123124 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:37.126136 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:37.126139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:37.126142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:37.126167 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:37.127742 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.128321 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.128327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.128333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.128411 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.128780 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.128784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.128785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.128988 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:37.129096 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:37.129651 (+ 555us) server_negotiation.cc:299] Negotiation successful
0923 17:19:37.129794 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:19:37.130914 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:37.116900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45879 (local address 127.23.214.129:53973)
0923 17:19:37.117454 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:37.117503 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:37.118893 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:37.119453 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:37.119460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:37.119867 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:37.120540 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:37.120558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.121511 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.121514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:37.122582 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:37.122600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.123172 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.123177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:37.123376 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:37.124072 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:37.124091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:37.125943 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:37.127892 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:37.127899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:37.127918 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:37.128203 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:37.128518 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:37.128521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:37.128523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:37.128678 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:37.129106 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:37.129115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:37.129398 (+ 283us) client_negotiation.cc:769] Sending connection context
0923 17:19:37.129599 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 17:19:37.130008 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":122,"threads_started":1}
I20260923 17:19:37.132411 27666 heartbeater.cc:347] Connected to a master server at 127.23.214.190:45879
I20260923 17:19:37.132696 27666 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:37.133373 27666 heartbeater.cc:511] Master 127.23.214.190:45879 requested a full tablet report, sending...
I20260923 17:19:37.135123 27475 ts_manager.cc:194] Registered new tserver with Master: a365d265fe31496790c13b4fa91be659 (127.23.214.129:42615)
I20260923 17:19:37.136484 27475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:53973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:37.153086 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45879
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:37.267486 27671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:37.267838 27671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:37.267917 27671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:37.272578 27671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:37.272680 27671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:37.272785 27671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:37.278532 27671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:37.279722 27671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:37.280812 27671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:37.291241 27676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:37.291280 27679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:37.291241 27677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:37.291527 27671 server_base.cc:1034] running on GCE node
I20260923 17:19:37.292119 27671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:37.292938 27671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:37.294142 27671 hybrid_clock.cc:648] HybridClock initialized: now 1790183977294120 us; error 34 us; skew 500 ppm
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:37.297122 27671 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:37.298141 27671 webserver.cc:460] Webserver started at http://127.23.214.130:43465/ using document root <none> and password file <none>
I20260923 17:19:37.298743 27671 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:37.298835 27671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:37.299117 27671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:37.301291 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "10af0a689f1849a0ad21e39ee43db48e"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "339eff2a83478b03e264fc12e2e06cc1"
server_key_iv: "7b39600e5f0608fb5c07ad6f87a4f3ca"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.301957 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "10af0a689f1849a0ad21e39ee43db48e"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "339eff2a83478b03e264fc12e2e06cc1"
server_key_iv: "7b39600e5f0608fb5c07ad6f87a4f3ca"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.306176 27671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:19:37.308847 27686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.310304 27671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:37.310446 27671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "10af0a689f1849a0ad21e39ee43db48e"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "339eff2a83478b03e264fc12e2e06cc1"
server_key_iv: "7b39600e5f0608fb5c07ad6f87a4f3ca"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.310573 27671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:37.353657 27671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:37.357184 27671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:37.357425 27671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:37.358194 27671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:37.359112 27671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:37.359190 27671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.359275 27671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:37.359329 27671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.368762 27671 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:40793
I20260923 17:19:37.368779 27799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:40793 every 8 connection(s)
I20260923 17:19:37.369756 27671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:37.370249 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27671
I20260923 17:19:37.370380 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:37.370671 24410 external_mini_cluster.cc:1428] Setting key 19b4d500a96da129c84ed638c8ca46eb
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:37.389166 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:37.372143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:58633 (local address 127.23.214.190:45879)
0923 17:19:37.372367 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:37.372370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:37.373256 (+ 886us) server_negotiation.cc:407] Connection header received
0923 17:19:37.375112 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:37.375116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:37.375180 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:37.375293 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:37.376603 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.377526 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.378868 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.379237 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.379368 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:37.384947 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:37.384954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:37.384960 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:37.385004 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:37.386682 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.387449 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.387453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.387457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.387551 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.387949 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.387952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.387954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.388159 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:37.388273 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:37.388845 (+ 572us) server_negotiation.cc:299] Negotiation successful
0923 17:19:37.388993 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:19:37.390190 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:37.372536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45879 (local address 127.23.214.130:58633)
0923 17:19:37.373067 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:37.373151 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:37.374899 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:37.375510 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:37.375518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:37.375906 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:37.376441 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:37.376455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.377670 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.377673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:37.378719 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:37.378729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.379412 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.379414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:37.379561 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:37.380235 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:37.380254 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:37.384675 (+ 4421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:37.386918 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:37.386924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:37.386941 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:37.387303 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:37.387659 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:37.387662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:37.387664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:37.387832 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:37.388272 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:37.388281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:37.388578 (+ 297us) client_negotiation.cc:769] Sending connection context
0923 17:19:37.388786 (+ 208us) client_negotiation.cc:240] Negotiation successful
0923 17:19:37.389171 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260923 17:19:37.391507 27800 heartbeater.cc:347] Connected to a master server at 127.23.214.190:45879
I20260923 17:19:37.391774 27800 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:37.392316 27800 heartbeater.cc:511] Master 127.23.214.190:45879 requested a full tablet report, sending...
I20260923 17:19:37.393445 27475 ts_manager.cc:194] Registered new tserver with Master: 10af0a689f1849a0ad21e39ee43db48e (127.23.214.130:40793)
I20260923 17:19:37.394167 27475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:58633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:37.402318 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45879
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:37.517279 27805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:37.517566 27805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:37.517658 27805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:37.521860 27805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:37.521957 27805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:37.522020 27805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:37.527427 27805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:37.528662 27805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:37.529686 27805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:37.539726 27810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:37.539740 27813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:37.539742 27811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:37.540086 27805 server_base.cc:1034] running on GCE node
I20260923 17:19:37.540557 27805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:37.541410 27805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:37.542603 27805 hybrid_clock.cc:648] HybridClock initialized: now 1790183977542531 us; error 82 us; skew 500 ppm
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:37.545673 27805 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:37.546628 27805 webserver.cc:460] Webserver started at http://127.23.214.131:43225/ using document root <none> and password file <none>
I20260923 17:19:37.547187 27805 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:37.547246 27805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:37.547490 27805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:37.549618 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "329361cc90a34225ad7874ff33414d20"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "fbd0109885e2bc1986839b760eed99b8"
server_key_iv: "993fb8dd8137284078a39ab3d44a46af"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.550209 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "329361cc90a34225ad7874ff33414d20"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "fbd0109885e2bc1986839b760eed99b8"
server_key_iv: "993fb8dd8137284078a39ab3d44a46af"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.554111 27805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:37.556811 27820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.558403 27805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:37.558540 27805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "329361cc90a34225ad7874ff33414d20"
format_stamp: "Formatted at 2026-09-23 17:19:37 on dist-test-slave-9q7f"
server_key: "fbd0109885e2bc1986839b760eed99b8"
server_key_iv: "993fb8dd8137284078a39ab3d44a46af"
server_key_version: "encryptionkey@0"
I20260923 17:19:37.558660 27805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:37.575742 27805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:37.579465 27805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:37.579703 27805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:37.580379 27805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:37.581270 27805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:37.581319 27805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.581391 27805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:37.581434 27805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:37.590920 27805 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:39005
I20260923 17:19:37.590962 27933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:39005 every 8 connection(s)
I20260923 17:19:37.591912 27805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:37.599116 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27805
I20260923 17:19:37.599269 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:37.599566 24410 external_mini_cluster.cc:1428] Setting key d1fa3ab2afc89633aca9b15c24c7b392
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:37.606599 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:37.593775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:39665 (local address 127.23.214.190:45879)
0923 17:19:37.593955 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:37.593960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:37.594826 (+ 866us) server_negotiation.cc:407] Connection header received
0923 17:19:37.596347 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:37.596351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:37.596418 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:37.596537 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:37.597937 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.598519 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.599976 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.600329 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.600466 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:37.603285 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:37.603289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:37.603291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:37.603319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:37.604681 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.605163 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.605166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.605167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.605240 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.605565 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.605568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.605569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.605772 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:37.605856 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:37.606372 (+ 516us) server_negotiation.cc:299] Negotiation successful
0923 17:19:37.606476 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:19:37.607465 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:37.594175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45879 (local address 127.23.214.131:39665)
0923 17:19:37.594639 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:37.594686 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:37.596106 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:37.596746 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:37.596759 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:37.597193 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:37.597718 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:37.597741 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.598657 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.598661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:37.599843 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:37.599852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.600544 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.600550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:37.600726 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:37.601423 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:37.601440 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:37.603084 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:37.604789 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:37.604794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:37.604815 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:37.605073 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:37.605354 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:37.605356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:37.605357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:37.605486 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:37.605860 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:37.605865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:37.606152 (+ 287us) client_negotiation.cc:769] Sending connection context
0923 17:19:37.606316 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 17:19:37.606579 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260923 17:19:37.608691 27934 heartbeater.cc:347] Connected to a master server at 127.23.214.190:45879
I20260923 17:19:37.608942 27934 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:37.609472 27934 heartbeater.cc:511] Master 127.23.214.190:45879 requested a full tablet report, sending...
I20260923 17:19:37.610579 27475 ts_manager.cc:194] Registered new tserver with Master: 329361cc90a34225ad7874ff33414d20 (127.23.214.131:39005)
I20260923 17:19:37.611158 27475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:39665
I20260923 17:19:37.613814 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183977, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:37.640408 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:37.631060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.23.214.190:45879)
0923 17:19:37.631237 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:37.631244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:37.631392 (+ 148us) server_negotiation.cc:407] Connection header received
0923 17:19:37.631536 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:37.631538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:37.631583 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:37.631654 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:37.632372 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.632937 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.634019 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:37.634242 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:37.634415 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:37.637111 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:37.637115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:37.637117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:37.637147 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:37.638685 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.639142 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.639145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.639147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.639244 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:37.639513 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:37.639515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:37.639517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:37.639768 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:37.639876 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:37.640069 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:19:37.640172 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260923 17:19:37.640890 27475 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44722
I20260923 17:19:37.641834 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27537
I20260923 17:19:37.647420 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27671
I20260923 17:19:37.653529 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27805
I20260923 17:19:37.659157 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27443
2026-09-23T17:19:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2222 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27946](info): set up 2 sockets
Sep 23 17:19:37 dist-test-slave-9q7f krb5kdc[27946](info): commencing operation
krb5kdc: starting...
W20260923 17:19:38.768530 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:38 dist-test-slave-9q7f krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:38.898164 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:41943
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:41943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:39.015836 27959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:39.016188 27959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:39.016234 27959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:39.020376 27959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:39.020465 27959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:39.020489 27959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:39.020509 27959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:39.020527 27959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:39.026115 27959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:41943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:39.027235 27959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:39.028298 27959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:39.041345 27964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:39.041345 27967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:39.041401 27965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:39.041833 27959 server_base.cc:1034] running on GCE node
I20260923 17:19:39.042623 27959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:39.044080 27959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:39.045301 27959 hybrid_clock.cc:648] HybridClock initialized: now 1790183979045273 us; error 48 us; skew 500 ppm
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:39.048243 27959 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:39.049219 27959 webserver.cc:460] Webserver started at http://127.23.214.190:40813/ using document root <none> and password file <none>
I20260923 17:19:39.049841 27959 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:39.050012 27959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:39.050261 27959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:39.052470 27959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "0ea78b03313c459e82e2d702ec9571be"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "8272bc15ff2d1157cfc3115be61d62bf"
server_key_iv: "6f99a4f590e004d8592a3dd5ec865754"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.053082 27959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "0ea78b03313c459e82e2d702ec9571be"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "8272bc15ff2d1157cfc3115be61d62bf"
server_key_iv: "6f99a4f590e004d8592a3dd5ec865754"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.057217 27959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260923 17:19:39.060093 27974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.061527 27959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:39.061694 27959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "0ea78b03313c459e82e2d702ec9571be"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "8272bc15ff2d1157cfc3115be61d62bf"
server_key_iv: "6f99a4f590e004d8592a3dd5ec865754"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.061822 27959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:39.098894 27959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:39.103672 27959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:39.103968 27959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:39.113833 27959 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:41943
I20260923 17:19:39.113859 28026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:41943 every 8 connection(s)
I20260923 17:19:39.115103 27959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:39.115671 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 27959
I20260923 17:19:39.115779 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:39.116082 24410 external_mini_cluster.cc:1428] Setting key a858963fd5073b7de5e93b71cc374895
I20260923 17:19:39.119333 28027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:39.125753 28027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be: Bootstrap starting.
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:39.128511 28027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:39.129323 28027 log.cc:824] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:39.131816 28027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be: No bootstrap required, opened a new log
I20260923 17:19:39.132344 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.117729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.23.214.190:41943)
0923 17:19:39.118412 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:39.118431 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:39.118476 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:19:39.119230 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:39.119251 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:39.119752 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:39.120173 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:39.121148 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.122386 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.123373 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.123921 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.124113 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.127127 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:39.127133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:39.127148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:39.127183 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:39.129108 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.129702 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.129708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.129713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.129922 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.130251 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.130256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.130258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.130679 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:39.130839 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:39.131180 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 17:19:39.131516 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":157,"threads_started":1}
I20260923 17:19:39.136076 28027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea78b03313c459e82e2d702ec9571be" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 41943 } }
I20260923 17:19:39.136399 28027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:39.136487 28027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea78b03313c459e82e2d702ec9571be, State: Initialized, Role: FOLLOWER
I20260923 17:19:39.136950 28027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea78b03313c459e82e2d702ec9571be" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 41943 } }
I20260923 17:19:39.137120 28027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:39.137212 28027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:39.137343 28027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:39.138504 28027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea78b03313c459e82e2d702ec9571be" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 41943 } }
I20260923 17:19:39.138900 28027 leader_election.cc:302] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ea78b03313c459e82e2d702ec9571be; no voters:
I20260923 17:19:39.139283 28027 leader_election.cc:288] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:39.139495 28032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:39.139758 28032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [term 1 LEADER]: Becoming Leader. State: Replica: 0ea78b03313c459e82e2d702ec9571be, State: Running, Role: LEADER
I20260923 17:19:39.140157 28032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea78b03313c459e82e2d702ec9571be" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 41943 } }
I20260923 17:19:39.140801 28027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:39.142495 28033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ea78b03313c459e82e2d702ec9571be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea78b03313c459e82e2d702ec9571be" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 41943 } } }
I20260923 17:19:39.142495 28034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea78b03313c459e82e2d702ec9571be. Latest consensus state: current_term: 1 leader_uuid: "0ea78b03313c459e82e2d702ec9571be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea78b03313c459e82e2d702ec9571be" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 41943 } } }
I20260923 17:19:39.142674 28033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:39.142680 28034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be [sys.catalog]: This master's current role is: LEADER
W20260923 17:19:39.146196 28047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:19:39.146306 28047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:19:39.146406 28046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:39.147243 28046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:39.153362 28046 catalog_manager.cc:1347] Generated new cluster ID: 65b4d0720dfe4395a6dc78ea39ec2136
I20260923 17:19:39.153452 28046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:39.170401 28046 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:39.171882 28046 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:39.181492 28046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ea78b03313c459e82e2d702ec9571be: Generated new TSK 0
I20260923 17:19:39.182456 28046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:39.218693 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:41943
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:39.334600 28053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:39.334898 28053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:39.334944 28053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:39.339102 28053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:39.339177 28053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:39.339239 28053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:39.345078 28053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:39.346221 28053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:39.347223 28053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:39.357726 28058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:39.357784 28059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:39.357766 28061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:39.358174 28053 server_base.cc:1034] running on GCE node
I20260923 17:19:39.358681 28053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:39.359378 28053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:39.360551 28053 hybrid_clock.cc:648] HybridClock initialized: now 1790183979360533 us; error 44 us; skew 500 ppm
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:39.363768 28053 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:39.365098 28053 webserver.cc:460] Webserver started at http://127.23.214.129:35077/ using document root <none> and password file <none>
I20260923 17:19:39.365794 28053 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:39.365933 28053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:39.366227 28053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:39.368649 28053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "be6aa9643b21485f81c6e1b45ec450d2"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "3a26ef90044d1b23f4b0716477accb3e"
server_key_iv: "617fda4d1bbb30186c60afb8946ccedd"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.369371 28053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "be6aa9643b21485f81c6e1b45ec450d2"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "3a26ef90044d1b23f4b0716477accb3e"
server_key_iv: "617fda4d1bbb30186c60afb8946ccedd"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.374138 28053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260923 17:19:39.376981 28068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.378386 28053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:39.378538 28053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "be6aa9643b21485f81c6e1b45ec450d2"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "3a26ef90044d1b23f4b0716477accb3e"
server_key_iv: "617fda4d1bbb30186c60afb8946ccedd"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.378669 28053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:39.397666 28053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:39.401335 28053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:39.401554 28053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:39.402256 28053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:39.403127 28053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:39.403208 28053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.403301 28053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:39.403359 28053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.412547 28053 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:43261
I20260923 17:19:39.412566 28181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:43261 every 8 connection(s)
I20260923 17:19:39.413532 28053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:39.415624 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 28053
I20260923 17:19:39.415738 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:39.416052 24410 external_mini_cluster.cc:1428] Setting key 100cc5ba2e673109de9a5b4e5d86e114
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:39.430570 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.415595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:40613 (local address 127.23.214.190:41943)
0923 17:19:39.415745 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:39.415752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:39.416899 (+ 1147us) server_negotiation.cc:407] Connection header received
0923 17:19:39.418502 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:39.418506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:39.418560 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:39.418639 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:39.419861 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.420401 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.422308 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.422556 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.422663 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.426052 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:39.426056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:39.426058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:39.426084 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:39.427665 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.428288 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.428291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.428293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.428380 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.428703 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.428706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.428707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.428964 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:39.429044 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:39.430184 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0923 17:19:39.430360 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:19:39.431605 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.416141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:41943 (local address 127.23.214.129:40613)
0923 17:19:39.416737 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:39.416795 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:39.418285 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:39.418823 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:39.418830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:39.419196 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:39.419694 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:39.419707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.420537 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.420544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:39.422175 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:39.422190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.422753 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.422769 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:39.423023 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.423771 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:39.423801 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:39.425797 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:39.427876 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:39.427882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:39.427898 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:39.428179 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:39.428470 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:39.428472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:39.428473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:39.428617 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:39.429232 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:39.429238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:39.429764 (+ 526us) client_negotiation.cc:769] Sending connection context
0923 17:19:39.430110 (+ 346us) client_negotiation.cc:240] Negotiation successful
0923 17:19:39.430603 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":164,"threads_started":1}
I20260923 17:19:39.433177 28182 heartbeater.cc:347] Connected to a master server at 127.23.214.190:41943
I20260923 17:19:39.433449 28182 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:39.434041 28182 heartbeater.cc:511] Master 127.23.214.190:41943 requested a full tablet report, sending...
I20260923 17:19:39.435942 27989 ts_manager.cc:194] Registered new tserver with Master: be6aa9643b21485f81c6e1b45ec450d2 (127.23.214.129:43261)
I20260923 17:19:39.437215 27989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:40613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:39.450373 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:41943
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:39.566135 28187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:39.566448 28187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:39.566533 28187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:39.571254 28187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:39.571331 28187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:39.571424 28187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:39.576921 28187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:39.578137 28187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:39.579222 28187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:39.589507 28192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:39.589556 28187 server_base.cc:1034] running on GCE node
W20260923 17:19:39.589493 28195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:39.589505 28193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:39.590194 28187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:39.590930 28187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:39.592098 28187 hybrid_clock.cc:648] HybridClock initialized: now 1790183979592062 us; error 50 us; skew 500 ppm
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:39.595064 28187 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:39.596113 28187 webserver.cc:460] Webserver started at http://127.23.214.130:44597/ using document root <none> and password file <none>
I20260923 17:19:39.596712 28187 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:39.596802 28187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:39.597054 28187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:39.599329 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "1ef741f8e8764a72863ce7dc5bd92764"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "b15e4fddda21e45e344f4eff4fca32d3"
server_key_iv: "4bd2bb409553ec2a477cf5d2f47d451a"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.599921 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "1ef741f8e8764a72863ce7dc5bd92764"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "b15e4fddda21e45e344f4eff4fca32d3"
server_key_iv: "4bd2bb409553ec2a477cf5d2f47d451a"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.603994 28187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:39.606798 28202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.608197 28187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:19:39.608312 28187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "1ef741f8e8764a72863ce7dc5bd92764"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "b15e4fddda21e45e344f4eff4fca32d3"
server_key_iv: "4bd2bb409553ec2a477cf5d2f47d451a"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.608414 28187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:39.635131 28187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:39.638499 28187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:39.638669 28187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:39.639227 28187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:39.640074 28187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:39.640123 28187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.640165 28187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:39.640180 28187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.649582 28187 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:33829
I20260923 17:19:39.649602 28315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:33829 every 8 connection(s)
I20260923 17:19:39.650645 28187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:39.657539 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 28187
I20260923 17:19:39.657677 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:39.658006 24410 external_mini_cluster.cc:1428] Setting key 9b7465f7f00bce741e6564d565e018f9
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:39.666131 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.652539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:49513 (local address 127.23.214.190:41943)
0923 17:19:39.652689 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:39.652693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:39.653596 (+ 903us) server_negotiation.cc:407] Connection header received
0923 17:19:39.655068 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:39.655071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:39.655121 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:39.655199 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:39.656452 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.657071 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.658781 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.659032 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.659125 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.661969 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:39.661972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:39.661974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:39.662000 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:39.663654 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.664266 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.664272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.664274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.664371 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.664789 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.664794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.664796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.665007 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:39.665127 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:39.665739 (+ 612us) server_negotiation.cc:299] Negotiation successful
0923 17:19:39.665927 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:19:39.667042 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.652967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:41943 (local address 127.23.214.130:49513)
0923 17:19:39.653435 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:39.653480 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:39.654850 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:39.655378 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:39.655385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:39.655781 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:39.656282 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:39.656296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.657304 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.657308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:39.658655 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:39.658664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.659200 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.659204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:39.659415 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.660062 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:39.660079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:39.661747 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:39.663816 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:39.663823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:39.663843 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:39.664139 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:39.664477 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:39.664482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:39.664484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:39.664680 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:39.665125 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:39.665133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:39.665418 (+ 285us) client_negotiation.cc:769] Sending connection context
0923 17:19:39.665681 (+ 263us) client_negotiation.cc:240] Negotiation successful
0923 17:19:39.666093 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:19:39.668366 28316 heartbeater.cc:347] Connected to a master server at 127.23.214.190:41943
I20260923 17:19:39.668654 28316 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:39.669140 28316 heartbeater.cc:511] Master 127.23.214.190:41943 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
I20260923 17:19:39.670416 27989 ts_manager.cc:194] Registered new tserver with Master: 1ef741f8e8764a72863ce7dc5bd92764 (127.23.214.130:33829)
I20260923 17:19:39.671284 27989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:49513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:39.688522 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:41943
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:39.804683 28321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:39.805035 28321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:39.805120 28321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:39.809295 28321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:39.809415 28321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:39.809523 28321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:39.815335 28321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:39.816532 28321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:39.817620 28321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:39.828222 28327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:39.828253 28329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:39.828248 28326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:39.828853 28321 server_base.cc:1034] running on GCE node
I20260923 17:19:39.829296 28321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:39.830077 28321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:39.831250 28321 hybrid_clock.cc:648] HybridClock initialized: now 1790183979831226 us; error 34 us; skew 500 ppm
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:39.834206 28321 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:39.835178 28321 webserver.cc:460] Webserver started at http://127.23.214.131:37661/ using document root <none> and password file <none>
I20260923 17:19:39.835762 28321 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:39.835865 28321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:39.836108 28321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:39.838397 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "b94a6afd367c4803ad0e73528db68948"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "a9d32c4440dc45839ad1c58dd9347617"
server_key_iv: "d3c04667ca2f49a225eae9971eba148a"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.839003 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "b94a6afd367c4803ad0e73528db68948"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "a9d32c4440dc45839ad1c58dd9347617"
server_key_iv: "d3c04667ca2f49a225eae9971eba148a"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.843046 28321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:39.845631 28336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.846930 28321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:19:39.847074 28321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "b94a6afd367c4803ad0e73528db68948"
format_stamp: "Formatted at 2026-09-23 17:19:39 on dist-test-slave-9q7f"
server_key: "a9d32c4440dc45839ad1c58dd9347617"
server_key_iv: "d3c04667ca2f49a225eae9971eba148a"
server_key_version: "encryptionkey@0"
I20260923 17:19:39.847194 28321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:39.864599 28321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:39.868199 28321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:39.868429 28321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:39.869092 28321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:39.870115 28321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:39.870193 28321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.870273 28321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:39.870332 28321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:39.880344 28321 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:46133
I20260923 17:19:39.880369 28449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:46133 every 8 connection(s)
I20260923 17:19:39.881361 28321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:39.885311 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 28321
I20260923 17:19:39.885517 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:39.885893 24410 external_mini_cluster.cc:1428] Setting key 83f9066e6af66fa9b0fbefa7f31e5c3d
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:39.896929 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.883706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:44849 (local address 127.23.214.190:41943)
0923 17:19:39.883859 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:39.883863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:39.884795 (+ 932us) server_negotiation.cc:407] Connection header received
0923 17:19:39.886565 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:39.886571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:39.886632 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:39.886708 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:39.888268 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.888809 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.890034 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.890258 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.890388 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.893132 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:39.893135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:39.893137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:39.893164 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:39.894796 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.895413 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.895416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.895418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.895500 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.895836 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.895838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.895839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.896031 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:39.896110 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:39.896688 (+ 578us) server_negotiation.cc:299] Negotiation successful
0923 17:19:39.896799 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:19:39.897917 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.884108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:41943 (local address 127.23.214.131:44849)
0923 17:19:39.884610 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:39.884654 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:39.886348 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:39.886915 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:39.886923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:39.887403 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:39.888124 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:39.888142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.888940 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.888943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:39.889845 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:39.889855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.890442 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.890446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:39.890627 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.891276 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:39.891292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:39.892942 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:39.894929 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:39.894935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:39.894949 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:39.895257 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:39.895587 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:39.895589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:39.895590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:39.895752 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:39.896150 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:39.896156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:39.896465 (+ 309us) client_negotiation.cc:769] Sending connection context
0923 17:19:39.896643 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 17:19:39.896957 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":158,"threads_started":1}
I20260923 17:19:39.899245 28450 heartbeater.cc:347] Connected to a master server at 127.23.214.190:41943
I20260923 17:19:39.899518 28450 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:39.900038 28450 heartbeater.cc:511] Master 127.23.214.190:41943 requested a full tablet report, sending...
I20260923 17:19:39.901090 27989 ts_manager.cc:194] Registered new tserver with Master: b94a6afd367c4803ad0e73528db68948 (127.23.214.131:46133)
I20260923 17:19:39.901754 27989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:44849
I20260923 17:19:39.912171 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:39.939455 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.929546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.23.214.129:43261)
0923 17:19:39.929843 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:39.929861 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:39.929941 (+ 80us) server_negotiation.cc:407] Connection header received
0923 17:19:39.930026 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:39.930032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:39.930242 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:39.930387 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:39.931106 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.931937 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.932879 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.933264 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.933350 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.935922 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:39.935931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:39.935945 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:39.935971 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:39.937495 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.937968 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.937974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.937976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.938056 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.938335 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.938338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.938340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.938664 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:39.938901 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:39.939115 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 17:19:39.939279 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
Sep 23 17:19:39 dist-test-slave-9q7f krb5kdc[27946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:39.950865 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:39.942675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.23.214.190:41943)
0923 17:19:39.942824 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:39.942828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:39.942844 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:19:39.942950 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:39.942952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:39.942991 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:39.943056 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:39.943714 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.944235 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.944991 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:39.945193 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:39.945266 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:39.947828 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:39.947832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:39.947834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:39.947859 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:39.949232 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.949625 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.949628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.949629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.949702 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:39.950026 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:39.950028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:39.950029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:39.950264 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:39.950379 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:39.950625 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:19:39.950724 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 17:19:39.951364 27989 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51268
I20260923 17:19:39.952286 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 28053
I20260923 17:19:39.958429 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 28187
I20260923 17:19:39.964381 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 28321
I20260923 17:19:39.969856 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 27959
2026-09-23T17:19:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2310 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:40 dist-test-slave-9q7f krb5kdc[28462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:40 dist-test-slave-9q7f krb5kdc[28462](info): set up 2 sockets
Sep 23 17:19:40 dist-test-slave-9q7f krb5kdc[28462](info): commencing operation
krb5kdc: starting...
W20260923 17:19:41.028293 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:41 dist-test-slave-9q7f krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:41.156947 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38203
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:38203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:41.268993 28475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:41.269353 28475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:41.269447 28475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:41.273663 28475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:41.273766 28475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:41.273806 28475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:41.273840 28475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:41.273895 28475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:41.279448 28475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:38203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:41.280616 28475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:41.281692 28475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:41.290210 28483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:41.290234 28481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:41.290287 28475 server_base.cc:1034] running on GCE node
W20260923 17:19:41.290205 28480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:41.291047 28475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:41.292212 28475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:41.293393 28475 hybrid_clock.cc:648] HybridClock initialized: now 1790183981293380 us; error 26 us; skew 500 ppm
Sep 23 17:19:41 dist-test-slave-9q7f krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183981, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:41.300606 28475 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:41.301375 28475 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:41.301445 28475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:41.301620 28475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:41.303857 28475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "4fda3e2bed224bc6b07597ed66693298"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "a1810227f3e17f674be3bfd4daa85808"
server_key_iv: "5e90c9f261ad68332a1952edd47cfe45"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.304493 28475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "4fda3e2bed224bc6b07597ed66693298"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "a1810227f3e17f674be3bfd4daa85808"
server_key_iv: "5e90c9f261ad68332a1952edd47cfe45"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.308656 28475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:41.311352 28489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:41.312668 28475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:19:41.312808 28475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "4fda3e2bed224bc6b07597ed66693298"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "a1810227f3e17f674be3bfd4daa85808"
server_key_iv: "5e90c9f261ad68332a1952edd47cfe45"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.312924 28475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:41.338498 28475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:41.342093 28475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:41.342263 28475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:41.349934 28475 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:38203
I20260923 17:19:41.350224 28541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:38203 every 8 connection(s)
I20260923 17:19:41.350725 28475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:41.353477 28542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:41.353999 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 28475
I20260923 17:19:41.354137 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:41.354411 24410 external_mini_cluster.cc:1428] Setting key 8bab280dd9cb554d61c995fef0827222
I20260923 17:19:41.360522 28542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298: Bootstrap starting.
I20260923 17:19:41.363070 28542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:41.363827 28542 log.cc:824] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:19:41 dist-test-slave-9q7f krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:41.366053 28542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298: No bootstrap required, opened a new log
I20260923 17:19:41.370137 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:41.355926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.23.214.190:38203)
0923 17:19:41.356389 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:41.356406 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:41.356444 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:19:41.357248 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:41.357282 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:41.357657 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:41.358106 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:41.359043 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.360156 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.361180 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.361709 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.361997 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:41.364789 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:41.364795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:41.364809 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:41.364837 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:41.366733 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:41.367328 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:41.367367 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:41.367375 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:41.367635 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:41.368035 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:41.368038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:41.368042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:41.368448 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:41.368604 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:41.368911 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 17:19:41.369220 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":121,"threads_started":1}
I20260923 17:19:41.370556 28542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda3e2bed224bc6b07597ed66693298" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38203 } }
I20260923 17:19:41.370815 28542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:41.370893 28542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3e2bed224bc6b07597ed66693298, State: Initialized, Role: FOLLOWER
I20260923 17:19:41.371362 28542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda3e2bed224bc6b07597ed66693298" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38203 } }
I20260923 17:19:41.371515 28542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:41.371580 28542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:41.371670 28542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:41.372809 28542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda3e2bed224bc6b07597ed66693298" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38203 } }
I20260923 17:19:41.373095 28542 leader_election.cc:302] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fda3e2bed224bc6b07597ed66693298; no voters:
I20260923 17:19:41.373342 28542 leader_election.cc:288] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:41.373461 28547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:41.373746 28547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3e2bed224bc6b07597ed66693298, State: Running, Role: LEADER
I20260923 17:19:41.374100 28547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda3e2bed224bc6b07597ed66693298" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38203 } }
I20260923 17:19:41.374635 28542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:41.375854 28549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fda3e2bed224bc6b07597ed66693298. Latest consensus state: current_term: 1 leader_uuid: "4fda3e2bed224bc6b07597ed66693298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda3e2bed224bc6b07597ed66693298" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38203 } } }
I20260923 17:19:41.375869 28548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fda3e2bed224bc6b07597ed66693298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda3e2bed224bc6b07597ed66693298" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38203 } } }
I20260923 17:19:41.376065 28548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:41.376030 28549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:41.376508 28556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:41.379951 28556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:41.390256 28556 catalog_manager.cc:1347] Generated new cluster ID: 4629489027204e40bed81ab6dd8860b4
I20260923 17:19:41.390353 28556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:41.407701 28556 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:41.408798 28556 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:41.418048 28556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298: Generated new TSK 0
I20260923 17:19:41.419011 28556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:41.457659 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38203
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:41.572379 28568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:41.572674 28568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:41.572722 28568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:41.577119 28568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:41.577185 28568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:41.577287 28568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:41.582840 28568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:41.584008 28568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:41.585093 28568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:41.594522 28576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:41.594550 28574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:41.594720 28568 server_base.cc:1034] running on GCE node
W20260923 17:19:41.594530 28573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:41.595296 28568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:41.595971 28568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:41.597187 28568 hybrid_clock.cc:648] HybridClock initialized: now 1790183981597171 us; error 29 us; skew 500 ppm
Sep 23 17:19:41 dist-test-slave-9q7f krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183981, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:41.600263 28568 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:41.601064 28568 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:41.601132 28568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:41.601302 28568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:41.603416 28568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "d2753d7585b64ecc8a54968d3e5bed27"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "86dbe446a26fa4fe54489b22c06de09a"
server_key_iv: "65c30421bfa352d56fe1b79778a37d92"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.603950 28568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "d2753d7585b64ecc8a54968d3e5bed27"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "86dbe446a26fa4fe54489b22c06de09a"
server_key_iv: "65c30421bfa352d56fe1b79778a37d92"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.607882 28568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:41.610658 28582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:41.612010 28568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:41.612156 28568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "d2753d7585b64ecc8a54968d3e5bed27"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "86dbe446a26fa4fe54489b22c06de09a"
server_key_iv: "65c30421bfa352d56fe1b79778a37d92"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.612285 28568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:41.643205 28568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:41.647403 28568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:41.647677 28568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:41.648452 28568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:41.649451 28568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:41.649507 28568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:41.649590 28568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:41.649629 28568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:41.660880 28568 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:41437
I20260923 17:19:41.660902 28695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:41437 every 8 connection(s)
I20260923 17:19:41.661641 28568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:41.665352 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 28568
I20260923 17:19:41.665467 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:41.665835 24410 external_mini_cluster.cc:1428] Setting key acf1ce6c88458ed47e62b108ea47cab0
Sep 23 17:19:41 dist-test-slave-9q7f krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183981, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:19:41.676887 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:41.663588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:53783 (local address 127.23.214.190:38203)
0923 17:19:41.663737 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:41.663741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:41.665020 (+ 1279us) server_negotiation.cc:407] Connection header received
0923 17:19:41.666133 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:41.666136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:41.666203 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:41.666293 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:41.667701 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.668237 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.669426 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.669666 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.669782 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:41.672700 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:41.672704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:41.672706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:41.672731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:41.674393 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:41.675067 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:41.675076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:41.675077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:41.675155 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:41.675642 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:41.675646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:41.675648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:41.675861 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:41.675992 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:41.676619 (+ 627us) server_negotiation.cc:299] Negotiation successful
0923 17:19:41.676741 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:41.678138 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:41.663969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38203 (local address 127.23.214.129:53783)
0923 17:19:41.664850 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:41.664894 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:41.665845 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:41.666491 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:41.666497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:41.666997 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:41.667554 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:41.667567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.668403 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.668406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:41.669291 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:41.669300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.669827 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.669831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:41.670088 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:41.670739 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:41.670756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:41.672508 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:41.674621 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:41.674628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:41.674645 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:41.674923 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:41.675251 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:41.675256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:41.675258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:41.675472 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:41.675990 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:41.675997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:41.676333 (+ 336us) client_negotiation.cc:769] Sending connection context
0923 17:19:41.676555 (+ 222us) client_negotiation.cc:240] Negotiation successful
0923 17:19:41.676888 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":129,"threads_started":1}
I20260923 17:19:41.679512 28696 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38203
I20260923 17:19:41.679769 28696 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:41.680290 28696 heartbeater.cc:511] Master 127.23.214.190:38203 requested a full tablet report, sending...
I20260923 17:19:41.681854 28506 ts_manager.cc:194] Registered new tserver with Master: d2753d7585b64ecc8a54968d3e5bed27 (127.23.214.129:41437)
I20260923 17:19:41.683092 28506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:53783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:41.695366 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38203
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:41.807631 28701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:41.807925 28701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:41.807973 28701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:41.812074 28701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:41.812139 28701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:41.812201 28701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:41.817866 28701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:41.819001 28701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:41.820026 28701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:41.829517 28707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:41.829521 28709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:41.829867 28701 server_base.cc:1034] running on GCE node
W20260923 17:19:41.829517 28706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:41.830451 28701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:41.831162 28701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:41.832357 28701 hybrid_clock.cc:648] HybridClock initialized: now 1790183981832322 us; error 48 us; skew 500 ppm
Sep 23 17:19:41 dist-test-slave-9q7f krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183981, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:41.835302 28701 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:41.836020 28701 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:41.836084 28701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:41.836258 28701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:41.838404 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "19ab1c80145e4143b4a61653bcde962e"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "459afcc2ddd1d0917cb039000c20e68a"
server_key_iv: "277f7f1a3c39f3e422296a7e91138dee"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.838932 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "19ab1c80145e4143b4a61653bcde962e"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "459afcc2ddd1d0917cb039000c20e68a"
server_key_iv: "277f7f1a3c39f3e422296a7e91138dee"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.842888 28701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:19:41.845700 28715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:41.847146 28701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:41.847254 28701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "19ab1c80145e4143b4a61653bcde962e"
format_stamp: "Formatted at 2026-09-23 17:19:41 on dist-test-slave-9q7f"
server_key: "459afcc2ddd1d0917cb039000c20e68a"
server_key_iv: "277f7f1a3c39f3e422296a7e91138dee"
server_key_version: "encryptionkey@0"
I20260923 17:19:41.847361 28701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:41.876531 28701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:41.880089 28701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:41.880249 28701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:41.880788 28701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:41.881587 28701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:41.881634 28701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:41.881675 28701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:41.881697 28701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:41.890772 28701 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:39771
I20260923 17:19:41.890808 28828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:39771 every 8 connection(s)
I20260923 17:19:41.891496 28701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:41.892366 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 28701
I20260923 17:19:41.892457 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:41.892720 24410 external_mini_cluster.cc:1428] Setting key 6fb0d6e8f7fbfabb569a132a260acca0
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Sep 23 17:19:41 dist-test-slave-9q7f krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183981, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:41.913172 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:41.894714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:52961 (local address 127.23.214.190:38203)
0923 17:19:41.894934 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:41.894939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:41.896585 (+ 1646us) server_negotiation.cc:407] Connection header received
0923 17:19:41.897493 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:41.897497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:41.897561 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:41.897668 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:41.899503 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.900035 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.902045 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.902301 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.902436 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:41.908284 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:41.908292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:41.908298 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:41.908342 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:41.910246 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:41.911145 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:41.911148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:41.911150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:41.911241 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:41.911783 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:41.911786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:41.911787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:41.912007 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:41.912105 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:41.912915 (+ 810us) server_negotiation.cc:299] Negotiation successful
0923 17:19:41.913040 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260923 17:19:41.914577 28831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:41.895295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38203 (local address 127.23.214.130:52961)
0923 17:19:41.896367 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:41.896444 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:41.897271 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:41.897870 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:41.897912 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:41.898478 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:41.899260 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:41.899310 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.900188 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.900194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:41.901846 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:41.901862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:41.902527 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:41.902533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:41.902779 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:41.903536 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:41.903561 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:41.908013 (+ 4452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:41.910429 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:41.910441 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:41.910466 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:41.910977 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:41.911356 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:41.911360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:41.911362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:41.911607 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:41.912131 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:41.912142 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:41.912570 (+ 428us) client_negotiation.cc:769] Sending connection context
0923 17:19:41.912836 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 17:19:41.913266 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":131,"threads_started":1}
I20260923 17:19:41.916181 28829 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38203
I20260923 17:19:41.916548 28829 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:41.917181 28829 heartbeater.cc:511] Master 127.23.214.190:38203 requested a full tablet report, sending...
I20260923 17:19:41.918581 28506 ts_manager.cc:194] Registered new tserver with Master: 19ab1c80145e4143b4a61653bcde962e (127.23.214.130:39771)
I20260923 17:19:41.919153 28506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:52961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:41.923295 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38203
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:42.039996 28834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:42.040292 28834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:42.040371 28834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:42.044549 28834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:42.044651 28834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:42.044747 28834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:42.050305 28834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:42.051517 28834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:42.052572 28834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:42.062425 28839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:42.062477 28840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:42.062605 28834 server_base.cc:1034] running on GCE node
W20260923 17:19:42.062438 28842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:42.063203 28834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:42.063926 28834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:42.065117 28834 hybrid_clock.cc:648] HybridClock initialized: now 1790183982065100 us; error 29 us; skew 500 ppm
Sep 23 17:19:42 dist-test-slave-9q7f krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183982, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:42.067999 28834 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:42.068737 28834 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:42.068831 28834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:42.069181 28834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:42.071753 28834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "554f0a1f30cd4758b8e6a7b397703451"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "a5a4d6377a59e7d9e6c0d8fae0e3131a"
server_key_iv: "8a19906c08805d7b1c4465c712c1e131"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.072364 28834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "554f0a1f30cd4758b8e6a7b397703451"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "a5a4d6377a59e7d9e6c0d8fae0e3131a"
server_key_iv: "8a19906c08805d7b1c4465c712c1e131"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.076535 28834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:42.079140 28848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:42.080539 28834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:42.080679 28834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "554f0a1f30cd4758b8e6a7b397703451"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "a5a4d6377a59e7d9e6c0d8fae0e3131a"
server_key_iv: "8a19906c08805d7b1c4465c712c1e131"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.080801 28834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:42.103048 28834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:42.106451 28834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:42.106663 28834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:42.107431 28834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:42.108300 28834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:42.108377 28834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:42.108443 28834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:42.108492 28834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:42.118203 28834 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:43707
I20260923 17:19:42.118252 28961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:43707 every 8 connection(s)
I20260923 17:19:42.118853 28834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:42.120659 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 28834
I20260923 17:19:42.120839 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:42.121248 24410 external_mini_cluster.cc:1428] Setting key 8f8efc1d5073cdf3cceaf2d0cac93930
Sep 23 17:19:42 dist-test-slave-9q7f krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183982, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:42.133661 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.120866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:39607 (local address 127.23.214.190:38203)
0923 17:19:42.121064 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.121070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.122631 (+ 1561us) server_negotiation.cc:407] Connection header received
0923 17:19:42.123544 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.123547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.123601 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.123676 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:42.124855 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.125408 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.126669 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.126900 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.127014 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.129831 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:42.129834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:42.129836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:42.129864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:42.131602 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:42.132114 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:42.132116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:42.132117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:42.132189 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:42.132544 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:42.132546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:42.132547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:42.132722 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:42.132813 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.133392 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.133527 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:19:42.134660 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.121325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38203 (local address 127.23.214.131:39607)
0923 17:19:42.122489 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.122532 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:42.123331 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:42.123855 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:42.123862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:42.124223 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:42.124715 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.124728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.125557 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.125560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.126541 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.126550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.127055 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.127060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.127235 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.127904 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:42.127924 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:42.129606 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:42.131720 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:42.131725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:42.131741 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:42.132022 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:42.132268 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:42.132271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:42.132272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:42.132460 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:42.132863 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:42.132869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:42.133133 (+ 264us) client_negotiation.cc:769] Sending connection context
0923 17:19:42.133326 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 17:19:42.133627 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":128,"threads_started":1}
I20260923 17:19:42.135921 28962 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38203
I20260923 17:19:42.136189 28962 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:42.136662 28962 heartbeater.cc:511] Master 127.23.214.190:38203 requested a full tablet report, sending...
I20260923 17:19:42.137844 28505 ts_manager.cc:194] Registered new tserver with Master: 554f0a1f30cd4758b8e6a7b397703451 (127.23.214.131:43707)
I20260923 17:19:42.138622 28505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:39607
I20260923 17:19:42.147418 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:19:42.158267 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.149501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.23.214.190:38203)
0923 17:19:42.149694 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.149700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.149894 (+ 194us) server_negotiation.cc:407] Connection header received
0923 17:19:42.150113 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.150117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.150179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.150324 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:42.151272 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.151964 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.152957 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.153222 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.153357 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.154964 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:42.154968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:42.154972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:42.155007 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:42.156341 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:42.156812 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:42.156819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:42.156821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:42.156911 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:42.157206 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:42.157211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:42.157213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:42.157498 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:42.157657 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.157936 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.158069 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:19:42.161950 28505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:42.165334 28505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:42.181279 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.174585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:41437 (local address 127.0.0.1:57192)
0923 17:19:42.175559 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.175613 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:42.175828 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:42.176658 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:42.176666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:42.176696 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:42.176851 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.176859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.178106 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.178113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.179489 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.179499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.180733 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.180737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.180862 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.180902 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 17:19:42.181015 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:19:42.181136 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":1,"spinlock_wait_cycles":4352,"thread_start_us":155,"threads_started":1}
I20260923 17:19:42.181355 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.174685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.23.214.129:41437)
0923 17:19:42.175333 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.175341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.175706 (+ 365us) server_negotiation.cc:407] Connection header received
0923 17:19:42.176052 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.176058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.176317 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.176453 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:42.176988 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.177952 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.179623 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.180591 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.180739 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.180895 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.181066 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.181183 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":88,"threads_started":1}
I20260923 17:19:42.181469 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.175456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:39771 (local address 127.0.0.1:42170)
0923 17:19:42.176074 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.176087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:42.176157 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:42.176809 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:42.176811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:42.176821 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:42.176900 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.176904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.178438 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.178443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.179650 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.179660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.180967 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.180972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.181116 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.181141 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:19:42.181216 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:19:42.181306 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260923 17:19:42.181953 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.175644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.23.214.130:39771)
0923 17:19:42.176413 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.176424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.176446 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:19:42.176516 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.176521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.176696 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.176844 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:42.177011 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.178277 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.179792 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.180619 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.180739 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.180895 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.181648 (+ 753us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.181760 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":109,"threads_started":1}
I20260923 17:19:42.184773 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.174870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.23.214.131:43707)
0923 17:19:42.175305 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.175318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.175880 (+ 562us) server_negotiation.cc:407] Connection header received
0923 17:19:42.176136 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.176142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.176352 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.176476 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:42.177201 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.178373 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.180114 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.180710 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.180843 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.180975 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.184490 (+ 3515us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.184621 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":118,"threads_started":1}
I20260923 17:19:42.184491 28630 tablet_service.cc:1467] Processing CreateTablet for tablet 13517dd4ada34b68af53d4a4e578b0ff (DEFAULT_TABLE table=test-table [id=f141ce8a2e2e4e628e6b9fd559464d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:42.184494 28763 tablet_service.cc:1467] Processing CreateTablet for tablet 13517dd4ada34b68af53d4a4e578b0ff (DEFAULT_TABLE table=test-table [id=f141ce8a2e2e4e628e6b9fd559464d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:42.185130 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.174617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:43707 (local address 127.0.0.1:49726)
0923 17:19:42.175773 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.175787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:42.175958 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:42.176657 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:42.176661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:42.176689 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:42.177076 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.177083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.178508 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.178511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.179996 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.180003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.180967 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.180972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.181692 (+ 720us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.181715 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:19:42.182335 (+ 620us) client_negotiation.cc:240] Negotiation successful
0923 17:19:42.182412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":1321,"spinlock_wait_cycles":3712,"thread_start_us":143,"threads_started":1}
I20260923 17:19:42.185715 28763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13517dd4ada34b68af53d4a4e578b0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:42.185716 28630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13517dd4ada34b68af53d4a4e578b0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:42.188285 28896 tablet_service.cc:1467] Processing CreateTablet for tablet 13517dd4ada34b68af53d4a4e578b0ff (DEFAULT_TABLE table=test-table [id=f141ce8a2e2e4e628e6b9fd559464d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:42.189481 28896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13517dd4ada34b68af53d4a4e578b0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:42.192083 28979 tablet_bootstrap.cc:492] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Bootstrap starting.
I20260923 17:19:42.193833 28980 tablet_bootstrap.cc:492] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27: Bootstrap starting.
I20260923 17:19:42.194641 28979 tablet_bootstrap.cc:654] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:42.195859 28979 log.cc:824] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:42.196348 28980 tablet_bootstrap.cc:654] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:42.197229 28980 log.cc:824] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:42.197561 28982 tablet_bootstrap.cc:492] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Bootstrap starting.
I20260923 17:19:42.198359 28979 tablet_bootstrap.cc:492] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: No bootstrap required, opened a new log
I20260923 17:19:42.198573 28979 ts_tablet_manager.cc:1397] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:19:42.199695 28982 tablet_bootstrap.cc:654] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:42.201471 28982 log.cc:824] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:42.201699 28979 raft_consensus.cc:348] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.202032 28979 raft_consensus.cc:374] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:42.202104 28979 raft_consensus.cc:729] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ab1c80145e4143b4a61653bcde962e, State: Initialized, Role: FOLLOWER
I20260923 17:19:42.202548 28979 consensus_queue.cc:260] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.203190 28829 heartbeater.cc:503] Master 127.23.214.190:38203 was elected leader, sending a full tablet report...
I20260923 17:19:42.203220 28980 tablet_bootstrap.cc:492] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27: No bootstrap required, opened a new log
I20260923 17:19:42.203447 28980 ts_tablet_manager.cc:1397] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260923 17:19:42.203562 28982 tablet_bootstrap.cc:492] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: No bootstrap required, opened a new log
I20260923 17:19:42.203775 28982 ts_tablet_manager.cc:1397] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:19:42.204680 28979 ts_tablet_manager.cc:1428] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260923 17:19:42.208190 28982 raft_consensus.cc:348] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.208549 28982 raft_consensus.cc:374] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:42.208617 28982 raft_consensus.cc:729] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554f0a1f30cd4758b8e6a7b397703451, State: Initialized, Role: FOLLOWER
I20260923 17:19:42.208858 28980 raft_consensus.cc:348] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.209131 28980 raft_consensus.cc:374] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:42.209203 28980 raft_consensus.cc:729] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2753d7585b64ecc8a54968d3e5bed27, State: Initialized, Role: FOLLOWER
I20260923 17:19:42.209132 28982 consensus_queue.cc:260] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.209673 28980 consensus_queue.cc:260] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.209892 28962 heartbeater.cc:503] Master 127.23.214.190:38203 was elected leader, sending a full tablet report...
I20260923 17:19:42.210404 28696 heartbeater.cc:503] Master 127.23.214.190:38203 was elected leader, sending a full tablet report...
I20260923 17:19:42.210738 28980 ts_tablet_manager.cc:1428] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 17:19:42.211345 28982 ts_tablet_manager.cc:1428] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260923 17:19:42.339773 28987 raft_consensus.cc:482] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:42.340052 28987 raft_consensus.cc:504] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.341269 28987 leader_election.cc:288] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554f0a1f30cd4758b8e6a7b397703451 (127.23.214.131:43707), 19ab1c80145e4143b4a61653bcde962e (127.23.214.130:39771)
I20260923 17:19:42.345516 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.341613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:43707 (local address 127.23.214.129:57311)
0923 17:19:42.342125 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.342140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:42.342248 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:42.342507 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:42.342509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:42.342522 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:42.342644 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.342651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.343405 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.343413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.344362 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.344370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.345034 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.345038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.345199 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.345218 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:19:42.345281 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 17:19:42.345337 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":118,"thread_start_us":123,"threads_started":1}
I20260923 17:19:42.345582 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.341667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:57311 (local address 127.23.214.131:43707)
0923 17:19:42.341826 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.341831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.342262 (+ 431us) server_negotiation.cc:407] Connection header received
0923 17:19:42.342328 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.342352 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.342405 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.342477 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:42.342767 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.343265 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.344493 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.344902 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.345041 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.345112 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.345337 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.345425 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:19:42.345715 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.341703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:35717 (local address 127.23.214.130:39771)
0923 17:19:42.341850 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.341855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.342116 (+ 261us) server_negotiation.cc:407] Connection header received
0923 17:19:42.342219 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.342223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.342274 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.342369 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:42.342674 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.343259 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.344485 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.344895 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.345060 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.345121 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.345480 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.345568 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:19:42.346140 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.341605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:39771 (local address 127.23.214.129:35717)
0923 17:19:42.342015 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.342028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:42.342131 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:42.342389 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:42.342393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:42.342409 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:42.342553 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.342561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.343380 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.343386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.344361 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:42.344369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.345034 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.345038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:42.345345 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.345368 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:19:42.345442 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:19:42.345507 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":220,"thread_start_us":141,"threads_started":1}
I20260923 17:19:42.346279 28916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13517dd4ada34b68af53d4a4e578b0ff" candidate_uuid: "d2753d7585b64ecc8a54968d3e5bed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554f0a1f30cd4758b8e6a7b397703451" is_pre_election: true
I20260923 17:19:42.346591 28916 raft_consensus.cc:2454] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2753d7585b64ecc8a54968d3e5bed27 in term 0.
I20260923 17:19:42.346760 28783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13517dd4ada34b68af53d4a4e578b0ff" candidate_uuid: "d2753d7585b64ecc8a54968d3e5bed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ab1c80145e4143b4a61653bcde962e" is_pre_election: true
I20260923 17:19:42.347075 28783 raft_consensus.cc:2454] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2753d7585b64ecc8a54968d3e5bed27 in term 0.
I20260923 17:19:42.347098 28583 leader_election.cc:302] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554f0a1f30cd4758b8e6a7b397703451, d2753d7585b64ecc8a54968d3e5bed27; no voters:
I20260923 17:19:42.347397 28987 raft_consensus.cc:2781] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:42.347505 28987 raft_consensus.cc:482] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:42.347565 28987 raft_consensus.cc:3037] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:42.348742 28987 raft_consensus.cc:504] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.349128 28987 leader_election.cc:288] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [CANDIDATE]: Term 1 election: Requested vote from peers 554f0a1f30cd4758b8e6a7b397703451 (127.23.214.131:43707), 19ab1c80145e4143b4a61653bcde962e (127.23.214.130:39771)
I20260923 17:19:42.349464 28916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13517dd4ada34b68af53d4a4e578b0ff" candidate_uuid: "d2753d7585b64ecc8a54968d3e5bed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554f0a1f30cd4758b8e6a7b397703451"
I20260923 17:19:42.349625 28916 raft_consensus.cc:3037] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:42.349620 28783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13517dd4ada34b68af53d4a4e578b0ff" candidate_uuid: "d2753d7585b64ecc8a54968d3e5bed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ab1c80145e4143b4a61653bcde962e"
I20260923 17:19:42.349752 28783 raft_consensus.cc:3037] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:42.350898 28916 raft_consensus.cc:2454] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2753d7585b64ecc8a54968d3e5bed27 in term 1.
I20260923 17:19:42.351204 28583 leader_election.cc:302] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554f0a1f30cd4758b8e6a7b397703451, d2753d7585b64ecc8a54968d3e5bed27; no voters:
I20260923 17:19:42.351251 28783 raft_consensus.cc:2454] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2753d7585b64ecc8a54968d3e5bed27 in term 1.
I20260923 17:19:42.351405 28987 raft_consensus.cc:2781] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:42.351666 28987 raft_consensus.cc:686] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [term 1 LEADER]: Becoming Leader. State: Replica: d2753d7585b64ecc8a54968d3e5bed27, State: Running, Role: LEADER
I20260923 17:19:42.352016 28987 consensus_queue.cc:237] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } }
I20260923 17:19:42.355350 28505 catalog_manager.cc:5495] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2753d7585b64ecc8a54968d3e5bed27 (127.23.214.129). New cstate: current_term: 1 leader_uuid: "d2753d7585b64ecc8a54968d3e5bed27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2753d7585b64ecc8a54968d3e5bed27" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 41437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 } health_report { overall_health: UNKNOWN } } }
W20260923 17:19:42.369845 28963 tablet.cc:2369] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:42.376012 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.371924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.23.214.129:41437)
0923 17:19:42.372057 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.372060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.372074 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:19:42.372218 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.372221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.372276 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.372389 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:42.372692 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.373190 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.374229 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:42.374545 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:42.374626 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:42.375017 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:42.375570 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:42.375664 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.375759 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.375810 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:19:42.382347 28783 raft_consensus.cc:1261] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 1 FOLLOWER]: Refusing update from remote peer d2753d7585b64ecc8a54968d3e5bed27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:42.382364 28916 raft_consensus.cc:1261] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 1 FOLLOWER]: Refusing update from remote peer d2753d7585b64ecc8a54968d3e5bed27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:42.383104 28987 consensus_queue.cc:1035] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554f0a1f30cd4758b8e6a7b397703451" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:42.383318 28990 consensus_queue.cc:1035] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ab1c80145e4143b4a61653bcde962e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:42.401365 28505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:42.401532 28505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:42.404165 28505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fda3e2bed224bc6b07597ed66693298: Sending DeleteTablet for 3 replicas of tablet 13517dd4ada34b68af53d4a4e578b0ff
I20260923 17:19:42.405021 28896 tablet_service.cc:1514] Processing DeleteTablet for tablet 13517dd4ada34b68af53d4a4e578b0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:42 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:49726
I20260923 17:19:42.405220 28763 tablet_service.cc:1514] Processing DeleteTablet for tablet 13517dd4ada34b68af53d4a4e578b0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:42 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:42170
I20260923 17:19:42.405309 28630 tablet_service.cc:1514] Processing DeleteTablet for tablet 13517dd4ada34b68af53d4a4e578b0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:42 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:57192
I20260923 17:19:42.405647 29003 tablet_replica.cc:330] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: stopping tablet replica
I20260923 17:19:42.406047 29003 raft_consensus.cc:2229] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:42.406126 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 28568
I20260923 17:19:42.406365 29004 tablet_replica.cc:330] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: stopping tablet replica
I20260923 17:19:42.406399 29003 raft_consensus.cc:2258] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:42.406530 29005 tablet_replica.cc:330] T 13517dd4ada34b68af53d4a4e578b0ff P d2753d7585b64ecc8a54968d3e5bed27: stopping tablet replica
I20260923 17:19:42.407001 29004 raft_consensus.cc:2229] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:42.407389 29004 raft_consensus.cc:2258] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:42.409216 29004 ts_tablet_manager.cc:1905] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:42.409966 29003 ts_tablet_manager.cc:1905] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:42.411520 29004 ts_tablet_manager.cc:1918] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:42.411634 29004 log.cc:1196] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/13517dd4ada34b68af53d4a4e578b0ff
I20260923 17:19:42.411918 29004 ts_tablet_manager.cc:1939] T 13517dd4ada34b68af53d4a4e578b0ff P 19ab1c80145e4143b4a61653bcde962e: Deleting consensus metadata
I20260923 17:19:42.413112 28492 catalog_manager.cc:4841] TS 19ab1c80145e4143b4a61653bcde962e (127.23.214.130:39771): tablet 13517dd4ada34b68af53d4a4e578b0ff (table test-table [id=f141ce8a2e2e4e628e6b9fd559464d02]) successfully deleted
I20260923 17:19:42.413697 29003 ts_tablet_manager.cc:1918] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:42.413810 29003 log.cc:1196] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/13517dd4ada34b68af53d4a4e578b0ff
I20260923 17:19:42.414201 29003 ts_tablet_manager.cc:1939] T 13517dd4ada34b68af53d4a4e578b0ff P 554f0a1f30cd4758b8e6a7b397703451: Deleting consensus metadata
W20260923 17:19:42.414752 28492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:41437 (error 108)
I20260923 17:19:42.415580 28490 catalog_manager.cc:4841] TS 554f0a1f30cd4758b8e6a7b397703451 (127.23.214.131:43707): tablet 13517dd4ada34b68af53d4a4e578b0ff (table test-table [id=f141ce8a2e2e4e628e6b9fd559464d02]) successfully deleted
I20260923 17:19:42.415871 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 28701
I20260923 17:19:42.415904 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.415552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:41437 (local address 127.0.0.1:57206)
0923 17:19:42.415731 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.415812 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:41437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 17:19:42.416494 28492 catalog_manager.cc:4584] TS d2753d7585b64ecc8a54968d3e5bed27 (127.23.214.129:41437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13517dd4ada34b68af53d4a4e578b0ff: Network error: Client connection negotiation failed: client connection to 127.23.214.129:41437: connect: Connection refused (error 111)
I20260923 17:19:42.422904 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 28834
I20260923 17:19:42.428615 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 28475
2026-09-23T17:19:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2459 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T17:19:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:42Z Disabled control of system clock
I20260923 17:19:42.462075 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40149
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:19:42.579746 29010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:42.580055 29010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:42.580096 29010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:42.584451 29010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:42.584532 29010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:42.584549 29010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:42.584568 29010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:42.584583 29010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:42.589761 29010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:40149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:42.590999 29010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:42.592444 29010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:42.602221 29015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:42.602234 29016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:42.602236 29018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:42.602617 29010 server_base.cc:1034] running on GCE node
I20260923 17:19:42.603261 29010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:42.604420 29010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:42.605605 29010 hybrid_clock.cc:648] HybridClock initialized: now 1790183982605566 us; error 52 us; skew 500 ppm
I20260923 17:19:42.608376 29010 webserver.cc:460] Webserver started at http://127.23.214.190:43127/ using document root <none> and password file <none>
I20260923 17:19:42.609012 29010 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:42.609112 29010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:42.609382 29010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:42.611690 29010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "d464a6bb3a3f431bb19a31a541fef445"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "328e36dfe0ae1c58148948072d8b6ece"
server_key_iv: "18ceb4367f7d0c8c59e2f08a4d4b38b8"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.612264 29010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "d464a6bb3a3f431bb19a31a541fef445"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "328e36dfe0ae1c58148948072d8b6ece"
server_key_iv: "18ceb4367f7d0c8c59e2f08a4d4b38b8"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.616305 29010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 17:19:42.619233 29024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:42.620699 29010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:42.620868 29010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "d464a6bb3a3f431bb19a31a541fef445"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "328e36dfe0ae1c58148948072d8b6ece"
server_key_iv: "18ceb4367f7d0c8c59e2f08a4d4b38b8"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.621026 29010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:42.626439 29010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:42.627166 29010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:42.627393 29010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:42.634945 29010 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:40149
I20260923 17:19:42.634968 29076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:40149 every 8 connection(s)
I20260923 17:19:42.636222 29010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:42.638682 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29010
I20260923 17:19:42.638834 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:42.639158 24410 external_mini_cluster.cc:1428] Setting key 18a41cf5ca8436723ea3622d07a144e4
I20260923 17:19:42.640448 29077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:42.646111 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.640654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.23.214.190:40149)
0923 17:19:42.641422 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.641445 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.641491 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:19:42.642483 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.642508 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.643015 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.643462 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:42.644293 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:42.644299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:42.644316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:42.644328 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:42.644402 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:42.644428 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:42.644606 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.644700 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.644992 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
I20260923 17:19:42.648382 29077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445: Bootstrap starting.
I20260923 17:19:42.650934 29077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:42.651953 29077 log.cc:824] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:42.654243 29077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445: No bootstrap required, opened a new log
I20260923 17:19:42.658107 29077 raft_consensus.cc:348] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d464a6bb3a3f431bb19a31a541fef445" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40149 } }
I20260923 17:19:42.658401 29077 raft_consensus.cc:374] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:42.658478 29077 raft_consensus.cc:729] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d464a6bb3a3f431bb19a31a541fef445, State: Initialized, Role: FOLLOWER
I20260923 17:19:42.659029 29077 consensus_queue.cc:260] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d464a6bb3a3f431bb19a31a541fef445" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40149 } }
I20260923 17:19:42.659246 29077 raft_consensus.cc:388] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:42.659332 29077 raft_consensus.cc:482] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:42.659441 29077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:42.660511 29077 raft_consensus.cc:504] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d464a6bb3a3f431bb19a31a541fef445" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40149 } }
I20260923 17:19:42.660893 29077 leader_election.cc:302] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d464a6bb3a3f431bb19a31a541fef445; no voters:
I20260923 17:19:42.661296 29077 leader_election.cc:288] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:42.661479 29082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:42.661741 29082 raft_consensus.cc:686] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [term 1 LEADER]: Becoming Leader. State: Replica: d464a6bb3a3f431bb19a31a541fef445, State: Running, Role: LEADER
I20260923 17:19:42.662086 29082 consensus_queue.cc:237] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d464a6bb3a3f431bb19a31a541fef445" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40149 } }
I20260923 17:19:42.662678 29077 sys_catalog.cc:564] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:42.663787 29084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d464a6bb3a3f431bb19a31a541fef445. Latest consensus state: current_term: 1 leader_uuid: "d464a6bb3a3f431bb19a31a541fef445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d464a6bb3a3f431bb19a31a541fef445" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40149 } } }
I20260923 17:19:42.663941 29084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:42.664245 29083 sys_catalog.cc:455] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d464a6bb3a3f431bb19a31a541fef445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d464a6bb3a3f431bb19a31a541fef445" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40149 } } }
I20260923 17:19:42.664361 29083 sys_catalog.cc:458] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:42.664448 29091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:42.667514 29091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:42.673135 29091 catalog_manager.cc:1347] Generated new cluster ID: 696834d0fed043e398e0d5f6bbb580fd
I20260923 17:19:42.673226 29091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:42.700299 29091 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:42.700690 29091 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:42.714167 29091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445: Generated new TSK 0
I20260923 17:19:42.715086 29091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:19:42.727519 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40149
--builtin_ntp_servers=127.23.214.148:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:19:42.850317 29101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:42.850628 29101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:42.850670 29101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:42.854897 29101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:42.854966 29101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:42.855073 29101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:42.860083 29101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:42.861337 29101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:42.863019 29101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:42.873848 29109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:42.873929 29107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:42.874045 29101 server_base.cc:1034] running on GCE node
W20260923 17:19:42.874353 29106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:42.874825 29101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:42.875655 29101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:42.876869 29101 hybrid_clock.cc:648] HybridClock initialized: now 1790183982876854 us; error 30 us; skew 500 ppm
I20260923 17:19:42.878988 29101 webserver.cc:460] Webserver started at http://127.23.214.129:33437/ using document root <none> and password file <none>
I20260923 17:19:42.879618 29101 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:42.879719 29101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:42.879956 29101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:42.882229 29101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "397867d5ac6c4af99054cc988f509cf6"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "99245cd9499debf366a2c405f6523d3e"
server_key_iv: "1267c148af1d1c24b1ef8c8b5e5429a5"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.882869 29101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "397867d5ac6c4af99054cc988f509cf6"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "99245cd9499debf366a2c405f6523d3e"
server_key_iv: "1267c148af1d1c24b1ef8c8b5e5429a5"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.887039 29101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:42.889991 29115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:42.891396 29101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:19:42.891597 29101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "397867d5ac6c4af99054cc988f509cf6"
format_stamp: "Formatted at 2026-09-23 17:19:42 on dist-test-slave-9q7f"
server_key: "99245cd9499debf366a2c405f6523d3e"
server_key_iv: "1267c148af1d1c24b1ef8c8b5e5429a5"
server_key_version: "encryptionkey@0"
I20260923 17:19:42.891757 29101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:42.896853 29101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:42.897526 29101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:42.897727 29101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:42.898402 29101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:42.899364 29101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:42.899434 29101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:42.899506 29101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:42.899550 29101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:42.908995 29101 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:42653
I20260923 17:19:42.909101 29228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:42653 every 8 connection(s)
I20260923 17:19:42.910130 29101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:42.914765 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29101
I20260923 17:19:42.914921 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:42.915197 24410 external_mini_cluster.cc:1428] Setting key b30e76f363b7c1d94c88ee2fdc781714
I20260923 17:19:42.917356 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40149
--builtin_ntp_servers=127.23.214.148:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:19:42.917474 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.912240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:59265 (local address 127.23.214.190:40149)
0923 17:19:42.912423 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:42.912428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:42.913649 (+ 1221us) server_negotiation.cc:407] Connection header received
0923 17:19:42.915342 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:42.915346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:42.915399 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:42.915527 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:42.916975 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:42.916978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:42.916981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:42.917000 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:42.917022 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:42.917030 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:42.917131 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:42.917289 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 17:19:42.917337 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:19:42.918501 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:42.912690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40149 (local address 127.23.214.129:59265)
0923 17:19:42.913474 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:42.913525 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:42.915074 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:42.915779 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:42.915790 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:42.915932 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:42.916665 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:42.916667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:42.916728 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:42.916819 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:42.917117 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:42.917126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:42.917132 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:19:42.917333 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 17:19:42.917612 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260923 17:19:42.919924 29229 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40149
I20260923 17:19:42.920200 29229 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:42.920666 29229 heartbeater.cc:511] Master 127.23.214.190:40149 requested a full tablet report, sending...
I20260923 17:19:42.922360 29041 ts_manager.cc:194] Registered new tserver with Master: 397867d5ac6c4af99054cc988f509cf6 (127.23.214.129:42653)
W20260923 17:19:43.032797 29232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:43.033265 29232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:43.033351 29232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:43.038352 29232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:43.038442 29232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:43.038544 29232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:43.043709 29232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:43.044883 29232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:43.046402 29232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:43.056536 29240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:43.056581 29232 server_base.cc:1034] running on GCE node
W20260923 17:19:43.056527 29237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:43.056527 29238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:43.057155 29232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:43.057833 29232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:43.059075 29232 hybrid_clock.cc:648] HybridClock initialized: now 1790183983059042 us; error 47 us; skew 500 ppm
I20260923 17:19:43.060984 29232 webserver.cc:460] Webserver started at http://127.23.214.130:42281/ using document root <none> and password file <none>
I20260923 17:19:43.061583 29232 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:43.061648 29232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:43.061915 29232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:43.064363 29232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "c4168fee646742febd90a44287623dd3"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "10e05e0d748786ffd24ce5f015b24ed6"
server_key_iv: "8082638eea6ed9739033ceffd780c6d8"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.064983 29232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "c4168fee646742febd90a44287623dd3"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "10e05e0d748786ffd24ce5f015b24ed6"
server_key_iv: "8082638eea6ed9739033ceffd780c6d8"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.068975 29232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 17:19:43.071969 29246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:43.073388 29232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:43.073575 29232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "c4168fee646742febd90a44287623dd3"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "10e05e0d748786ffd24ce5f015b24ed6"
server_key_iv: "8082638eea6ed9739033ceffd780c6d8"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.073715 29232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:43.078428 29232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:43.079075 29232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:43.079303 29232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:43.079849 29232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:43.080808 29232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:43.080892 29232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:43.080978 29232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:43.081029 29232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:43.089831 29232 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:36177
I20260923 17:19:43.089854 29359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:36177 every 8 connection(s)
I20260923 17:19:43.090834 29232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:43.094362 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29232
I20260923 17:19:43.094506 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:43.094792 24410 external_mini_cluster.cc:1428] Setting key 3aca74275eadacd5f866cfda3f9864fc
I20260923 17:19:43.097039 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40149
--builtin_ntp_servers=127.23.214.148:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:19:43.097648 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.092745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:50633 (local address 127.23.214.190:40149)
0923 17:19:43.092894 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.092898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.094109 (+ 1211us) server_negotiation.cc:407] Connection header received
0923 17:19:43.095756 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.095759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.095800 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.095883 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.097131 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.097134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.097137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.097152 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.097177 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.097188 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.097266 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.097469 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.097516 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:19:43.098637 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.093190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40149 (local address 127.23.214.130:50633)
0923 17:19:43.093949 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.093999 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.095507 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.096078 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.096091 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.096257 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:43.096876 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:43.096878 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:43.096920 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:43.096991 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:43.097311 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:43.097317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:43.097322 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:19:43.097505 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 17:19:43.097766 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260923 17:19:43.099924 29360 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40149
I20260923 17:19:43.100181 29360 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:43.100621 29360 heartbeater.cc:511] Master 127.23.214.190:40149 requested a full tablet report, sending...
I20260923 17:19:43.101832 29041 ts_manager.cc:194] Registered new tserver with Master: c4168fee646742febd90a44287623dd3 (127.23.214.130:36177)
W20260923 17:19:43.213851 29363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:43.214202 29363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:43.214246 29363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:43.218416 29363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:43.218479 29363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:43.218537 29363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:43.223464 29363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:43.224546 29363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:43.225966 29363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:43.236255 29368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:43.236361 29363 server_base.cc:1034] running on GCE node
W20260923 17:19:43.236251 29371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:43.236251 29369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:43.236946 29363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:43.237612 29363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:43.238771 29363 hybrid_clock.cc:648] HybridClock initialized: now 1790183983238734 us; error 48 us; skew 500 ppm
I20260923 17:19:43.240902 29363 webserver.cc:460] Webserver started at http://127.23.214.131:43829/ using document root <none> and password file <none>
I20260923 17:19:43.241506 29363 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:43.241569 29363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:43.241828 29363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:43.244108 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "5ed43c950728428ba1e91b0db7bb92f5"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "700ff29afbfa3ff174e107d2d649f887"
server_key_iv: "e5660f48caa093f5cac83c7ef62488ac"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.244648 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "5ed43c950728428ba1e91b0db7bb92f5"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "700ff29afbfa3ff174e107d2d649f887"
server_key_iv: "e5660f48caa093f5cac83c7ef62488ac"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.248449 29363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:19:43.251269 29377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:43.252765 29363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:43.252911 29363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "5ed43c950728428ba1e91b0db7bb92f5"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "700ff29afbfa3ff174e107d2d649f887"
server_key_iv: "e5660f48caa093f5cac83c7ef62488ac"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.253044 29363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:43.258010 29363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:43.258701 29363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:43.258898 29363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:43.259481 29363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:43.260439 29363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:43.260484 29363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:43.260550 29363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:43.260586 29363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:43.270134 29363 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:43605
I20260923 17:19:43.270164 29490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:43605 every 8 connection(s)
I20260923 17:19:43.271132 29363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:43.274185 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29363
I20260923 17:19:43.274329 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:43.274688 24410 external_mini_cluster.cc:1428] Setting key 5a25d8b0d1d015db5ecb2df8fc63d2ad
I20260923 17:19:43.278424 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.273151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:45373 (local address 127.23.214.190:40149)
0923 17:19:43.273347 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.273352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.274711 (+ 1359us) server_negotiation.cc:407] Connection header received
0923 17:19:43.276371 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.276374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.276409 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.276468 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.277859 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.277865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.277906 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.277922 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.277949 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.277960 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.278042 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.278270 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.278315 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 17:19:43.279428 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.273587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40149 (local address 127.23.214.131:45373)
0923 17:19:43.274520 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.274583 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.276120 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.276697 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.276707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.276839 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:43.277521 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:43.277523 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:43.277579 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:43.277654 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:43.278047 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:43.278054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:43.278059 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:19:43.278315 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 17:19:43.278575 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":126,"threads_started":1}
I20260923 17:19:43.280704 29491 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40149
I20260923 17:19:43.280970 29491 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:43.281483 29491 heartbeater.cc:511] Master 127.23.214.190:40149 requested a full tablet report, sending...
I20260923 17:19:43.282625 29041 ts_manager.cc:194] Registered new tserver with Master: 5ed43c950728428ba1e91b0db7bb92f5 (127.23.214.131:43605)
I20260923 17:19:43.289609 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:19:43.293493 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.291866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.23.214.190:40149)
0923 17:19:43.292000 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.292004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.292202 (+ 198us) server_negotiation.cc:407] Connection header received
0923 17:19:43.292356 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.292358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.292392 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.292461 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.293137 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.293140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.293142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.293153 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.293172 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.293180 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.293249 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.293354 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.293387 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:19:43.296877 29041 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:43.300051 29041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:43.312134 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.308576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:36177 (local address 127.0.0.1:34052)
0923 17:19:43.309315 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.309329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.309437 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.310301 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.310309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.310391 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:43.311040 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:43.311042 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:43.311065 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:43.311137 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:43.311746 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:43.311758 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:43.311767 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:19:43.311900 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 17:19:43.311973 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":107,"threads_started":1}
I20260923 17:19:43.312186 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.308963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.23.214.130:36177)
0923 17:19:43.309299 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.309325 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.309434 (+ 109us) server_negotiation.cc:407] Connection header received
0923 17:19:43.309537 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.309543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.310124 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.310691 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.311573 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.311577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.311586 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.311593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.311625 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.311639 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.311765 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.311926 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.312006 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
I20260923 17:19:43.312626 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.309598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.23.214.131:43605)
0923 17:19:43.310244 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.310272 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.310294 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:19:43.310341 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.310346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.310515 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.310662 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.311795 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.311801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.311811 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.311822 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.311887 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.311916 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.312015 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.312402 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.312475 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":304,"threads_started":1}
I20260923 17:19:43.312310 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.308508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:43605 (local address 127.0.0.1:42688)
0923 17:19:43.308873 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.308922 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.309025 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.310805 (+ 1780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.310810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.310827 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:43.311607 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:43.311609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:43.311617 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:43.311639 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:43.312058 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:43.312061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:43.312065 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:19:43.312133 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 17:19:43.312167 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":119,"threads_started":1}
I20260923 17:19:43.313181 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.309458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.23.214.129:42653)
0923 17:19:43.309860 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.309892 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.309916 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:19:43.310060 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.310066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.310250 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.310388 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.311964 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.311969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.311979 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.311990 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.312029 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.312057 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.312152 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.312949 (+ 797us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.313018 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":95,"threads_started":1}
I20260923 17:19:43.312465 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.309297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42653 (local address 127.0.0.1:38930)
0923 17:19:43.309781 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.309796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.309918 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.310571 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.310574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.310589 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:43.311790 (+ 1201us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:43.311792 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:43.311801 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:43.311820 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:43.312197 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:43.312200 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:43.312203 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 17:19:43.312280 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:19:43.312325 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":93,"threads_started":1}
I20260923 17:19:43.315598 29294 tablet_service.cc:1467] Processing CreateTablet for tablet 968933ef560348b58fb71950f0cc1b20 (DEFAULT_TABLE table=test-table [id=ad36f109ff7140338b4ee2f38ec32ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:43.315591 29425 tablet_service.cc:1467] Processing CreateTablet for tablet 968933ef560348b58fb71950f0cc1b20 (DEFAULT_TABLE table=test-table [id=ad36f109ff7140338b4ee2f38ec32ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:43.315937 29163 tablet_service.cc:1467] Processing CreateTablet for tablet 968933ef560348b58fb71950f0cc1b20 (DEFAULT_TABLE table=test-table [id=ad36f109ff7140338b4ee2f38ec32ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:43.316722 29425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968933ef560348b58fb71950f0cc1b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:43.316722 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968933ef560348b58fb71950f0cc1b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:43.316751 29163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968933ef560348b58fb71950f0cc1b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:43.323504 29508 tablet_bootstrap.cc:492] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Bootstrap starting.
I20260923 17:19:43.324509 29509 tablet_bootstrap.cc:492] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Bootstrap starting.
I20260923 17:19:43.324975 29510 tablet_bootstrap.cc:492] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: Bootstrap starting.
I20260923 17:19:43.326021 29508 tablet_bootstrap.cc:654] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:43.326686 29509 tablet_bootstrap.cc:654] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:43.326927 29508 log.cc:824] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:43.327318 29510 tablet_bootstrap.cc:654] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:43.327487 29509 log.cc:824] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:43.328141 29510 log.cc:824] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:43.329264 29508 tablet_bootstrap.cc:492] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: No bootstrap required, opened a new log
I20260923 17:19:43.329355 29509 tablet_bootstrap.cc:492] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: No bootstrap required, opened a new log
I20260923 17:19:43.329479 29508 ts_tablet_manager.cc:1397] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 17:19:43.329560 29509 ts_tablet_manager.cc:1397] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:19:43.330139 29510 tablet_bootstrap.cc:492] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: No bootstrap required, opened a new log
I20260923 17:19:43.330317 29510 ts_tablet_manager.cc:1397] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:19:43.333477 29509 raft_consensus.cc:348] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.333742 29509 raft_consensus.cc:374] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:43.333807 29509 raft_consensus.cc:729] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4168fee646742febd90a44287623dd3, State: Initialized, Role: FOLLOWER
I20260923 17:19:43.334303 29509 consensus_queue.cc:260] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.334172 29508 raft_consensus.cc:348] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.334453 29508 raft_consensus.cc:374] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:43.334514 29508 raft_consensus.cc:729] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed43c950728428ba1e91b0db7bb92f5, State: Initialized, Role: FOLLOWER
I20260923 17:19:43.334702 29510 raft_consensus.cc:348] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.334957 29510 raft_consensus.cc:374] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:43.334935 29508 consensus_queue.cc:260] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.335023 29510 raft_consensus.cc:729] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397867d5ac6c4af99054cc988f509cf6, State: Initialized, Role: FOLLOWER
I20260923 17:19:43.335140 29360 heartbeater.cc:503] Master 127.23.214.190:40149 was elected leader, sending a full tablet report...
I20260923 17:19:43.335391 29509 ts_tablet_manager.cc:1428] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:19:43.335474 29510 consensus_queue.cc:260] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.335911 29508 ts_tablet_manager.cc:1428] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:19:43.336107 29229 heartbeater.cc:503] Master 127.23.214.190:40149 was elected leader, sending a full tablet report...
I20260923 17:19:43.336369 29510 ts_tablet_manager.cc:1428] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:19:43.336457 29491 heartbeater.cc:503] Master 127.23.214.190:40149 was elected leader, sending a full tablet report...
W20260923 17:19:43.341770 29361 tablet.cc:2369] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:19:43.411415 29230 tablet.cc:2369] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:43.504648 29515 raft_consensus.cc:482] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:43.504907 29515 raft_consensus.cc:504] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.506172 29515 leader_election.cc:288] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4168fee646742febd90a44287623dd3 (127.23.214.130:36177), 397867d5ac6c4af99054cc988f509cf6 (127.23.214.129:42653)
I20260923 17:19:43.508306 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.506546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42653 (local address 127.23.214.131:47389)
0923 17:19:43.506699 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.506725 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.506823 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.507140 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.507143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.507159 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:43.507711 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:43.507716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:43.507750 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:43.507785 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:43.508046 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:43.508049 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:43.508053 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:19:43.508134 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:19:43.508175 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 17:19:43.508306 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.506636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:47389 (local address 127.23.214.129:42653)
0923 17:19:43.506778 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.506783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.506804 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:19:43.506971 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.506974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.507012 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.507117 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.507906 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.507909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.507914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.507930 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.507950 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.507958 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.508053 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.508139 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.508180 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:19:43.508769 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.506553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:36177 (local address 127.23.214.131:47439)
0923 17:19:43.507033 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.507044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.507161 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.507501 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.507503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.507516 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:43.508117 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:19:43.508119 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:43.508138 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:19:43.508162 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:43.508508 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:43.508512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:43.508515 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 17:19:43.508594 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:19:43.508626 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":115,"thread_start_us":146,"threads_started":1}
I20260923 17:19:43.508780 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.506732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:47439 (local address 127.23.214.130:36177)
0923 17:19:43.506867 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.506875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.507139 (+ 264us) server_negotiation.cc:407] Connection header received
0923 17:19:43.507283 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.507286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.507351 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.507438 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.508294 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.508298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.508301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.508315 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.508339 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.508348 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.508462 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.508608 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.508649 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:19:43.508999 29183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968933ef560348b58fb71950f0cc1b20" candidate_uuid: "5ed43c950728428ba1e91b0db7bb92f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397867d5ac6c4af99054cc988f509cf6" is_pre_election: true
I20260923 17:19:43.509390 29183 raft_consensus.cc:2454] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed43c950728428ba1e91b0db7bb92f5 in term 0.
I20260923 17:19:43.509409 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968933ef560348b58fb71950f0cc1b20" candidate_uuid: "5ed43c950728428ba1e91b0db7bb92f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4168fee646742febd90a44287623dd3" is_pre_election: true
I20260923 17:19:43.509673 29314 raft_consensus.cc:2454] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed43c950728428ba1e91b0db7bb92f5 in term 0.
I20260923 17:19:43.509991 29378 leader_election.cc:302] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397867d5ac6c4af99054cc988f509cf6, 5ed43c950728428ba1e91b0db7bb92f5; no voters:
I20260923 17:19:43.510331 29515 raft_consensus.cc:2781] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:43.510413 29515 raft_consensus.cc:482] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:43.510445 29515 raft_consensus.cc:3037] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:43.511544 29515 raft_consensus.cc:504] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.511951 29515 leader_election.cc:288] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [CANDIDATE]: Term 1 election: Requested vote from peers c4168fee646742febd90a44287623dd3 (127.23.214.130:36177), 397867d5ac6c4af99054cc988f509cf6 (127.23.214.129:42653)
I20260923 17:19:43.512321 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968933ef560348b58fb71950f0cc1b20" candidate_uuid: "5ed43c950728428ba1e91b0db7bb92f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4168fee646742febd90a44287623dd3"
I20260923 17:19:43.512354 29183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968933ef560348b58fb71950f0cc1b20" candidate_uuid: "5ed43c950728428ba1e91b0db7bb92f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397867d5ac6c4af99054cc988f509cf6"
I20260923 17:19:43.512457 29314 raft_consensus.cc:3037] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:43.512458 29183 raft_consensus.cc:3037] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:43.513764 29183 raft_consensus.cc:2454] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed43c950728428ba1e91b0db7bb92f5 in term 1.
I20260923 17:19:43.513764 29314 raft_consensus.cc:2454] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed43c950728428ba1e91b0db7bb92f5 in term 1.
I20260923 17:19:43.514073 29378 leader_election.cc:302] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397867d5ac6c4af99054cc988f509cf6, 5ed43c950728428ba1e91b0db7bb92f5; no voters:
I20260923 17:19:43.514269 29515 raft_consensus.cc:2781] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:43.514501 29515 raft_consensus.cc:686] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed43c950728428ba1e91b0db7bb92f5, State: Running, Role: LEADER
I20260923 17:19:43.514865 29515 consensus_queue.cc:237] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } }
I20260923 17:19:43.518359 29039 catalog_manager.cc:5495] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ed43c950728428ba1e91b0db7bb92f5 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "5ed43c950728428ba1e91b0db7bb92f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed43c950728428ba1e91b0db7bb92f5" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 } health_report { overall_health: UNKNOWN } } }
W20260923 17:19:43.522069 29492 tablet.cc:2369] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:43.551467 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.549784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.23.214.131:43605)
0923 17:19:43.549971 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.549975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.549994 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:19:43.550098 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.550101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.550163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.550271 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.550991 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.550995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.550998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.551010 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.551037 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.551046 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.551141 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.551245 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.551289 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:19:43.557844 29183 raft_consensus.cc:1261] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 1 FOLLOWER]: Refusing update from remote peer 5ed43c950728428ba1e91b0db7bb92f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:43.557914 29314 raft_consensus.cc:1261] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 1 FOLLOWER]: Refusing update from remote peer 5ed43c950728428ba1e91b0db7bb92f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:43.558642 29518 consensus_queue.cc:1035] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4168fee646742febd90a44287623dd3" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:43.558893 29515 consensus_queue.cc:1035] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397867d5ac6c4af99054cc988f509cf6" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:43.577184 29039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:43.577389 29039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:43.580129 29039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d464a6bb3a3f431bb19a31a541fef445: Sending DeleteTablet for 3 replicas of tablet 968933ef560348b58fb71950f0cc1b20
I20260923 17:19:43.581019 29425 tablet_service.cc:1514] Processing DeleteTablet for tablet 968933ef560348b58fb71950f0cc1b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:43 UTC) from {username='slave'} at 127.0.0.1:42688
I20260923 17:19:43.581023 29294 tablet_service.cc:1514] Processing DeleteTablet for tablet 968933ef560348b58fb71950f0cc1b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:43 UTC) from {username='slave'} at 127.0.0.1:34052
I20260923 17:19:43.581180 29163 tablet_service.cc:1514] Processing DeleteTablet for tablet 968933ef560348b58fb71950f0cc1b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:43 UTC) from {username='slave'} at 127.0.0.1:38930
I20260923 17:19:43.581658 29533 tablet_replica.cc:330] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6: stopping tablet replica
I20260923 17:19:43.581938 29532 tablet_replica.cc:330] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: stopping tablet replica
I20260923 17:19:43.582062 29533 raft_consensus.cc:2229] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:43.582257 29532 raft_consensus.cc:2229] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:43.582408 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29101
I20260923 17:19:43.582422 29533 raft_consensus.cc:2258] T 968933ef560348b58fb71950f0cc1b20 P 397867d5ac6c4af99054cc988f509cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:43.582636 29532 raft_consensus.cc:2258] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:43.583601 29531 tablet_replica.cc:330] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: stopping tablet replica
I20260923 17:19:43.583947 29531 raft_consensus.cc:2229] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:43.584313 29531 raft_consensus.cc:2258] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:43.585283 29532 ts_tablet_manager.cc:1905] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:43.585947 29531 ts_tablet_manager.cc:1905] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:43.588164 29531 ts_tablet_manager.cc:1918] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:43.588261 29532 ts_tablet_manager.cc:1918] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:43.588299 29531 log.cc:1196] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/968933ef560348b58fb71950f0cc1b20
I20260923 17:19:43.588361 29532 log.cc:1196] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/968933ef560348b58fb71950f0cc1b20
I20260923 17:19:43.588614 29531 ts_tablet_manager.cc:1939] T 968933ef560348b58fb71950f0cc1b20 P c4168fee646742febd90a44287623dd3: Deleting consensus metadata
I20260923 17:19:43.588719 29532 ts_tablet_manager.cc:1939] T 968933ef560348b58fb71950f0cc1b20 P 5ed43c950728428ba1e91b0db7bb92f5: Deleting consensus metadata
I20260923 17:19:43.589861 29025 catalog_manager.cc:4841] TS 5ed43c950728428ba1e91b0db7bb92f5 (127.23.214.131:43605): tablet 968933ef560348b58fb71950f0cc1b20 (table test-table [id=ad36f109ff7140338b4ee2f38ec32ed0]) successfully deleted
I20260923 17:19:43.589861 29028 catalog_manager.cc:4841] TS c4168fee646742febd90a44287623dd3 (127.23.214.130:36177): tablet 968933ef560348b58fb71950f0cc1b20 (table test-table [id=ad36f109ff7140338b4ee2f38ec32ed0]) successfully deleted
W20260923 17:19:43.595336 29025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:42653 (error 108)
I20260923 17:19:43.595507 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29232
I20260923 17:19:43.597373 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.596947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42653 (local address 127.0.0.1:38942)
0923 17:19:43.597175 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.597260 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:42653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260923 17:19:43.597559 29025 catalog_manager.cc:4584] TS 397867d5ac6c4af99054cc988f509cf6 (127.23.214.129:42653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 968933ef560348b58fb71950f0cc1b20: Network error: Client connection negotiation failed: client connection to 127.23.214.129:42653: connect: Connection refused (error 111)
I20260923 17:19:43.603721 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29363
I20260923 17:19:43.610098 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29010
2026-09-23T17:19:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1180 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 17:19:43.632118 24410 mini_oidc.cc:152] Starting JWKS server
I20260923 17:19:43.633715 24410 webserver.cc:460] Webserver started at https://127.0.0.1:45123/ using document root <none> and password file <none>
I20260923 17:19:43.633867 24410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:19:43.634212 24410 webserver.cc:460] Webserver started at http://127.0.0.1:43441/ using document root <none> and password file <none>
2026-09-23T17:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:43Z Disabled control of system clock
I20260923 17:19:43.643708 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:43535
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:43535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/testchainca.cert with env {}
W20260923 17:19:43.756156 29540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:43.756458 29540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:43.756501 29540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:43.760632 29540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:43.760699 29540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:43.760715 29540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:43.760732 29540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:43.760747 29540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:19:43.760764 29540 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:19:43.760778 29540 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45123/jwks.json
I20260923 17:19:43.766330 29540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:43535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:43.767448 29540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:43.768441 29540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:43.777194 29546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:43.777186 29545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:43.777455 29540 server_base.cc:1034] running on GCE node
W20260923 17:19:43.777180 29548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:43.778242 29540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:43.779404 29540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:43.780601 29540 hybrid_clock.cc:648] HybridClock initialized: now 1790183983780585 us; error 25 us; skew 500 ppm
I20260923 17:19:43.782783 29540 webserver.cc:460] Webserver started at http://127.23.214.190:46455/ using document root <none> and password file <none>
I20260923 17:19:43.783406 29540 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:43.783478 29540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:43.783725 29540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:43.785972 29540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "9ca174c9fac448b6b75c1187107bd951"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "8156c6a6b4dd253b40cc2e5c9539560c"
server_key_iv: "5547a55253e95e17bd7af96f6c6d7cbf"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.786607 29540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "9ca174c9fac448b6b75c1187107bd951"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "8156c6a6b4dd253b40cc2e5c9539560c"
server_key_iv: "5547a55253e95e17bd7af96f6c6d7cbf"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.790516 29540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260923 17:19:43.793090 29554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:43.794534 29540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:43.794682 29540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "9ca174c9fac448b6b75c1187107bd951"
format_stamp: "Formatted at 2026-09-23 17:19:43 on dist-test-slave-9q7f"
server_key: "8156c6a6b4dd253b40cc2e5c9539560c"
server_key_iv: "5547a55253e95e17bd7af96f6c6d7cbf"
server_key_version: "encryptionkey@0"
I20260923 17:19:43.794801 29540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:43.809939 29540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:43.810802 29540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:43.811013 29540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:43.818212 29540 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:43535
I20260923 17:19:43.818282 29606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:43535 every 8 connection(s)
I20260923 17:19:43.819245 29540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:43.820029 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29540
I20260923 17:19:43.820186 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:43.820508 24410 external_mini_cluster.cc:1428] Setting key ab7cec8c9ef70f116ae60476bf137c26
I20260923 17:19:43.823122 29607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:43.829716 29607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951: Bootstrap starting.
I20260923 17:19:43.830211 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.821954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.23.214.190:43535)
0923 17:19:43.822578 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.822593 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.822629 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:19:43.823349 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.823367 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.823754 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.824035 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:43.824409 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:43.825384 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:43.826659 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:43.827149 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:43.827326 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:43.828198 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:43.828203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:43.828216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:43.828671 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:43.828728 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:43.828749 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:43.828828 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:43.828998 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 17:19:43.829353 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":128,"threads_started":1}
I20260923 17:19:43.832319 29607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:43.833200 29607 log.cc:824] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:43.833329 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:19:43.835407 29607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951: No bootstrap required, opened a new log
I20260923 17:19:43.838924 29607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca174c9fac448b6b75c1187107bd951" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43535 } }
I20260923 17:19:43.839185 29607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:43.839268 29607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca174c9fac448b6b75c1187107bd951, State: Initialized, Role: FOLLOWER
I20260923 17:19:43.839689 29607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca174c9fac448b6b75c1187107bd951" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43535 } }
I20260923 17:19:43.839812 29607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:43.839854 29607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:43.839969 29607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:43.840907 29607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca174c9fac448b6b75c1187107bd951" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43535 } }
I20260923 17:19:43.841262 29607 leader_election.cc:302] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ca174c9fac448b6b75c1187107bd951; no voters:
I20260923 17:19:43.841647 29607 leader_election.cc:288] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:43.841735 29612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:43.841961 29612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca174c9fac448b6b75c1187107bd951, State: Running, Role: LEADER
I20260923 17:19:43.842259 29612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca174c9fac448b6b75c1187107bd951" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43535 } }
I20260923 17:19:43.843019 29607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:43.844019 29613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ca174c9fac448b6b75c1187107bd951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca174c9fac448b6b75c1187107bd951" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43535 } } }
I20260923 17:19:43.844141 29613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:43.844415 29614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca174c9fac448b6b75c1187107bd951. Latest consensus state: current_term: 1 leader_uuid: "9ca174c9fac448b6b75c1187107bd951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca174c9fac448b6b75c1187107bd951" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43535 } } }
I20260923 17:19:43.844527 29614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:43.844537 29621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:43.847947 29621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:43.854460 29621 catalog_manager.cc:1347] Generated new cluster ID: 553fe8145aba43138641bb41f174d39f
I20260923 17:19:43.854543 29621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:43.908047 29621 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:43.909057 29621 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:43.916770 29621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ca174c9fac448b6b75c1187107bd951: Generated new TSK 0
I20260923 17:19:43.917508 29621 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:19:43.954164 29637 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 17:19:43.954301 29637 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:19:43.954860 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:43.937186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.23.214.190:43535)
0923 17:19:43.937364 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:43.937369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:43.937567 (+ 198us) server_negotiation.cc:407] Connection header received
0923 17:19:43.937796 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:43.937798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:43.937855 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:43.938015 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:19:43.938386 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:43.938979 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:43.940299 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:43.940531 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:43.940617 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:43.941054 (+ 437us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:19:43.954496 (+ 13442us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:19:43.954744 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 17:19:43.954957 29610 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 17:19:43.956516 29636 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:43.936994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43535 (local address 127.0.0.1:33226)
0923 17:19:43.937442 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:43.937459 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:43.937610 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:43.938061 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:43.938065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:43.938075 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:19:43.938239 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:43.938246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:43.939108 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:43.939112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:43.940119 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:43.940131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:43.940663 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:43.940667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:43.940768 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:43.940832 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:19:43.954761 (+ 13929us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:19:43.955066 (+ 305us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.214.190:43535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.214.190:43535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 17:19:43.957763 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29540
2026-09-23T17:19:43Z chronyd exiting
I20260923 17:19:43.996140 24410 test_util.cc:175] -----------------------------------------------
I20260923 17:19:43.996232 24410 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790183952943234-24410-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (365 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 17:19:43.997259 24410 mini_oidc.cc:152] Starting JWKS server
I20260923 17:19:43.998189 24410 webserver.cc:460] Webserver started at https://127.0.0.1:36275/ using document root <none> and password file <none>
I20260923 17:19:43.998277 24410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:19:43.998478 24410 webserver.cc:460] Webserver started at http://127.0.0.1:43699/ using document root <none> and password file <none>
2026-09-23T17:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:43Z Disabled control of system clock
I20260923 17:19:44.007658 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34661
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:34661
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36275/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/testchainca.cert with env {}
W20260923 17:19:44.120947 29643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:44.121239 29643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:44.121281 29643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:44.125427 29643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:44.125499 29643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:44.125514 29643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:44.125531 29643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:44.125546 29643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:19:44.125562 29643 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:19:44.125576 29643 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36275/jwks.json
I20260923 17:19:44.131127 29643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:34661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36275/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:44.132205 29643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:44.133297 29643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:44.141675 29648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:44.141696 29651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:44.141682 29649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:44.142006 29643 server_base.cc:1034] running on GCE node
I20260923 17:19:44.142694 29643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:44.143873 29643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:44.145062 29643 hybrid_clock.cc:648] HybridClock initialized: now 1790183984145030 us; error 49 us; skew 500 ppm
I20260923 17:19:44.147199 29643 webserver.cc:460] Webserver started at http://127.23.214.190:35989/ using document root <none> and password file <none>
I20260923 17:19:44.147804 29643 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:44.147905 29643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:44.148137 29643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:44.150378 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "7f1c54be2eb447d8ad5616d42552e537"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "9f1ad06c3988bd82e9185e754087a868"
server_key_iv: "3f452a22ef7d5c96f0803a7ecffb96fd"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.150959 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "7f1c54be2eb447d8ad5616d42552e537"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "9f1ad06c3988bd82e9185e754087a868"
server_key_iv: "3f452a22ef7d5c96f0803a7ecffb96fd"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.154757 29643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:44.157433 29657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:44.158944 29643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:19:44.159111 29643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "7f1c54be2eb447d8ad5616d42552e537"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "9f1ad06c3988bd82e9185e754087a868"
server_key_iv: "3f452a22ef7d5c96f0803a7ecffb96fd"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.159250 29643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:44.190229 29643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:44.190975 29643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:44.191177 29643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:44.198395 29643 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:34661
I20260923 17:19:44.198457 29709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:34661 every 8 connection(s)
I20260923 17:19:44.199462 29643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:44.202845 29710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:44.203939 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29643
I20260923 17:19:44.204079 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:44.204420 24410 external_mini_cluster.cc:1428] Setting key b530fa4613a297a8c332745f6aad8242
I20260923 17:19:44.209436 29710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537: Bootstrap starting.
I20260923 17:19:44.212278 29710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:44.213135 29710 log.cc:824] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:44.213994 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:44.205926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.23.214.190:34661)
0923 17:19:44.206582 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:44.206598 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:44.206635 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:19:44.207296 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:44.207313 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:44.207714 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:44.208009 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:44.208356 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.209301 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.210529 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.211014 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.211141 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.212101 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:44.212106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:44.212118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:44.212578 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:44.212635 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:44.212655 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:44.212735 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:44.212881 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 17:19:44.213176 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260923 17:19:44.215153 29710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537: No bootstrap required, opened a new log
I20260923 17:19:44.217074 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:19:44.219607 29710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1c54be2eb447d8ad5616d42552e537" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34661 } }
I20260923 17:19:44.219842 29710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:44.219877 29710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1c54be2eb447d8ad5616d42552e537, State: Initialized, Role: FOLLOWER
I20260923 17:19:44.220235 29710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1c54be2eb447d8ad5616d42552e537" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34661 } }
I20260923 17:19:44.220345 29710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:44.220391 29710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:44.220461 29710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:44.221352 29710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1c54be2eb447d8ad5616d42552e537" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34661 } }
I20260923 17:19:44.221648 29710 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1c54be2eb447d8ad5616d42552e537; no voters:
I20260923 17:19:44.222066 29710 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:44.222213 29715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:44.222468 29715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [term 1 LEADER]: Becoming Leader. State: Replica: 7f1c54be2eb447d8ad5616d42552e537, State: Running, Role: LEADER
I20260923 17:19:44.222839 29715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1c54be2eb447d8ad5616d42552e537" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34661 } }
I20260923 17:19:44.223310 29710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:44.224835 29717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1c54be2eb447d8ad5616d42552e537. Latest consensus state: current_term: 1 leader_uuid: "7f1c54be2eb447d8ad5616d42552e537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1c54be2eb447d8ad5616d42552e537" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34661 } } }
I20260923 17:19:44.225191 29717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:44.225087 29716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1c54be2eb447d8ad5616d42552e537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1c54be2eb447d8ad5616d42552e537" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34661 } } }
I20260923 17:19:44.225754 29716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537 [sys.catalog]: This master's current role is: LEADER
W20260923 17:19:44.228235 29729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:19:44.228307 29729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:19:44.228417 29730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:44.229125 29730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:44.234917 29730 catalog_manager.cc:1347] Generated new cluster ID: 499fd23aed674c8e81a47c92df943347
I20260923 17:19:44.234992 29730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:44.246604 29730 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:44.247884 29730 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:44.257126 29730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1c54be2eb447d8ad5616d42552e537: Generated new TSK 0
I20260923 17:19:44.258170 29730 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:19:44.287319 29740 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 17:19:44.287439 29740 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:19:44.288247 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:44.271790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.23.214.190:34661)
0923 17:19:44.271924 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:44.271930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:44.272055 (+ 125us) server_negotiation.cc:407] Connection header received
0923 17:19:44.272215 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:44.272218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:44.272280 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:44.272410 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:19:44.272632 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.273490 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.274518 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.274890 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.275001 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.275312 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:19:44.287738 (+ 12426us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:19:44.288087 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 17:19:44.288285 29739 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:44.271681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34661 (local address 127.0.0.1:58048)
0923 17:19:44.271963 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:44.271976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:44.272074 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:44.272421 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:44.272423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:44.272430 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:19:44.272524 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:44.272528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.273635 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.273638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:44.274394 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:44.274404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.275020 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.275023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:44.275093 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.275117 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:19:44.287990 (+ 12873us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:19:44.288144 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.214.190:34661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
W20260923 17:19:44.288321 29713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260923 17:19:44.289034 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29643
2026-09-23T17:19:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 17:19:44.328907 24410 mini_oidc.cc:152] Starting JWKS server
I20260923 17:19:44.329934 24410 webserver.cc:460] Webserver started at https://127.0.0.1:34059/ using document root <none> and password file <none>
I20260923 17:19:44.330087 24410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:19:44.330371 24410 webserver.cc:460] Webserver started at http://127.0.0.1:35539/ using document root <none> and password file <none>
2026-09-23T17:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:44Z Disabled control of system clock
I20260923 17:19:44.339951 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44777
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:44777
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34059/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:19:44.452869 29746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:44.453191 29746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:44.453233 29746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:44.457484 29746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:44.457558 29746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:44.457573 29746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:44.457592 29746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:44.457607 29746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:19:44.457623 29746 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:19:44.457636 29746 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34059/jwks.json
I20260923 17:19:44.462565 29746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:44777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34059/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:44.463667 29746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:44.465093 29746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:44.473968 29751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:44.473980 29752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:44.474020 29754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:44.474197 29746 server_base.cc:1034] running on GCE node
I20260923 17:19:44.474951 29746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:44.476172 29746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:44.477394 29746 hybrid_clock.cc:648] HybridClock initialized: now 1790183984477357 us; error 50 us; skew 500 ppm
I20260923 17:19:44.479522 29746 webserver.cc:460] Webserver started at http://127.23.214.190:45569/ using document root <none> and password file <none>
I20260923 17:19:44.480154 29746 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:44.480257 29746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:44.480499 29746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:44.482697 29746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "4a99a4a1e925417cacd6284f2e492d03"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "2e1a19534f965510e8ed9ad9f20fda14"
server_key_iv: "cfe7cee99e17612ca60ec9e5788803b2"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.483278 29746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "4a99a4a1e925417cacd6284f2e492d03"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "2e1a19534f965510e8ed9ad9f20fda14"
server_key_iv: "cfe7cee99e17612ca60ec9e5788803b2"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.487064 29746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:19:44.489583 29760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:44.490867 29746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:19:44.491020 29746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "4a99a4a1e925417cacd6284f2e492d03"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "2e1a19534f965510e8ed9ad9f20fda14"
server_key_iv: "cfe7cee99e17612ca60ec9e5788803b2"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.491139 29746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:44.513230 29746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:44.514132 29746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:44.514357 29746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:44.522043 29746 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:44777
I20260923 17:19:44.522054 29812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:44777 every 8 connection(s)
I20260923 17:19:44.523103 29746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:44.526362 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29746
I20260923 17:19:44.526480 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:44.526522 29813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:44.526803 24410 external_mini_cluster.cc:1428] Setting key 0430337965bc7f3ac2c7b0f3d825f03e
I20260923 17:19:44.533465 29813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03: Bootstrap starting.
I20260923 17:19:44.536037 29813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:44.536732 29816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:44.528316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.23.214.190:44777)
0923 17:19:44.529002 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:44.529018 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:44.529055 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:19:44.529915 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:44.529949 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:44.530395 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:44.530806 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:44.531265 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.532243 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.533208 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.533759 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.533950 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.534922 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:44.534927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:44.534940 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:44.535338 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:44.535395 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:44.535415 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:44.535528 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:44.535679 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 17:19:44.535941 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260923 17:19:44.536828 29813 log.cc:824] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:44.539363 29813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03: No bootstrap required, opened a new log
I20260923 17:19:44.539745 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:19:44.543660 29813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a99a4a1e925417cacd6284f2e492d03" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44777 } }
I20260923 17:19:44.543900 29813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:44.543977 29813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a99a4a1e925417cacd6284f2e492d03, State: Initialized, Role: FOLLOWER
I20260923 17:19:44.544413 29813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a99a4a1e925417cacd6284f2e492d03" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44777 } }
I20260923 17:19:44.544566 29813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:44.544655 29813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:44.544778 29813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:44.545780 29813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a99a4a1e925417cacd6284f2e492d03" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44777 } }
I20260923 17:19:44.546165 29813 leader_election.cc:302] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a99a4a1e925417cacd6284f2e492d03; no voters:
I20260923 17:19:44.546479 29813 leader_election.cc:288] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:44.546638 29818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:44.546901 29818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [term 1 LEADER]: Becoming Leader. State: Replica: 4a99a4a1e925417cacd6284f2e492d03, State: Running, Role: LEADER
I20260923 17:19:44.547313 29818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a99a4a1e925417cacd6284f2e492d03" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44777 } }
I20260923 17:19:44.547807 29813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:44.549109 29820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a99a4a1e925417cacd6284f2e492d03. Latest consensus state: current_term: 1 leader_uuid: "4a99a4a1e925417cacd6284f2e492d03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a99a4a1e925417cacd6284f2e492d03" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44777 } } }
I20260923 17:19:44.549232 29820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:44.549183 29819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a99a4a1e925417cacd6284f2e492d03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a99a4a1e925417cacd6284f2e492d03" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44777 } } }
I20260923 17:19:44.549297 29819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:44.549655 29828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:44.552695 29828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:44.558290 29828 catalog_manager.cc:1347] Generated new cluster ID: 67c27247256f4cf7a40dae1ded319e6f
I20260923 17:19:44.558370 29828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:44.574621 29828 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:44.575891 29828 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:44.581144 29828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a99a4a1e925417cacd6284f2e492d03: Generated new TSK 0
I20260923 17:19:44.581961 29828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:19:44.601109 29816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:44.595793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.23.214.190:44777)
0923 17:19:44.595946 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:44.595951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:44.596101 (+ 150us) server_negotiation.cc:407] Connection header received
0923 17:19:44.596256 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:44.596259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:44.596324 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:44.596440 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:19:44.596670 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.597509 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.598481 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.598855 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.598961 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.599240 (+ 279us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:19:44.600733 (+ 1493us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34059/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:19:44.600971 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47170: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34059/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 17:19:44.601186 29816 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47170: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34059/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 17:19:44.603335 29842 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:44.595625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44777 (local address 127.0.0.1:47170)
0923 17:19:44.595997 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:44.596020 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:44.596129 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:44.596446 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:44.596448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:44.596455 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:19:44.596566 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:44.596571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.597638 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.597641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:44.598371 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:44.598380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.598972 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.598975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:44.599047 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.599070 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:19:44.600912 (+ 1842us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34059/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:19:44.603146 (+ 2234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.214.190:44777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34059/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":2008,"thread_start_us":102,"threads_started":1}
I20260923 17:19:44.604126 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29746
E20260923 17:19:44.613292 29843 webserver.cc:533] Webserver: sslize failed:
E20260923 17:19:44.613390 29843 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T17:19:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T17:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:44Z Disabled control of system clock
I20260923 17:19:44.651858 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42687
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:38201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:42687
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/testchainca.cert with env {}
W20260923 17:19:44.763414 29847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:44.763702 29847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:44.763742 29847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:44.767822 29847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:44.767894 29847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:44.767910 29847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:44.767928 29847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:44.767943 29847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:19:44.767961 29847 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:19:44.767973 29847 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 17:19:44.773521 29847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:38201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:42687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:44.774693 29847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:44.775684 29847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:44.784297 29852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:44.784305 29853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:44.784379 29855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:44.784538 29847 server_base.cc:1034] running on GCE node
I20260923 17:19:44.785271 29847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:44.786535 29847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:44.787720 29847 hybrid_clock.cc:648] HybridClock initialized: now 1790183984787696 us; error 52 us; skew 500 ppm
I20260923 17:19:44.789670 29847 webserver.cc:460] Webserver started at http://127.23.214.190:37039/ using document root <none> and password file <none>
I20260923 17:19:44.790340 29847 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:44.790446 29847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:44.790673 29847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:44.792712 29847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "d9d41ff9d14f4bac8e76c97bcc15df96"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "f76d65898241da0a4a74d6ab0e1665a4"
server_key_iv: "0c47eda5eaaee1cb1a885147c297088f"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.793269 29847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "d9d41ff9d14f4bac8e76c97bcc15df96"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "f76d65898241da0a4a74d6ab0e1665a4"
server_key_iv: "0c47eda5eaaee1cb1a885147c297088f"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.796981 29847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:19:44.799556 29861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:44.800832 29847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:19:44.800971 29847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "d9d41ff9d14f4bac8e76c97bcc15df96"
format_stamp: "Formatted at 2026-09-23 17:19:44 on dist-test-slave-9q7f"
server_key: "f76d65898241da0a4a74d6ab0e1665a4"
server_key_iv: "0c47eda5eaaee1cb1a885147c297088f"
server_key_version: "encryptionkey@0"
I20260923 17:19:44.801090 29847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:44.834205 29847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:44.834964 29847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:44.835177 29847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:44.842312 29847 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:42687
I20260923 17:19:44.842355 29913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:42687 every 8 connection(s)
I20260923 17:19:44.843411 29847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:44.846359 29914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:44.847936 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29847
I20260923 17:19:44.848071 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:44.848376 24410 external_mini_cluster.cc:1428] Setting key dd474fa3a86bf020605efc81243c4f8e
I20260923 17:19:44.852742 29914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96: Bootstrap starting.
I20260923 17:19:44.855521 29914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:44.856328 29914 log.cc:824] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:44.857510 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:44.849651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.23.214.190:42687)
0923 17:19:44.850290 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:44.850307 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:44.850342 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:19:44.850962 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:44.850979 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:44.851372 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:44.851659 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:44.851992 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.852994 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.854198 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.854656 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.854772 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.855652 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:44.855663 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:44.855675 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:19:44.856059 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:44.856115 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:19:44.856134 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:44.856211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:44.856383 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 17:19:44.856645 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260923 17:19:44.858408 29914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96: No bootstrap required, opened a new log
I20260923 17:19:44.860303 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:19:44.862622 29914 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42687 } }
I20260923 17:19:44.862847 29914 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:44.862881 29914 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d41ff9d14f4bac8e76c97bcc15df96, State: Initialized, Role: FOLLOWER
I20260923 17:19:44.863234 29914 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42687 } }
I20260923 17:19:44.863341 29914 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:44.863385 29914 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:44.863451 29914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:44.864338 29914 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42687 } }
I20260923 17:19:44.864629 29914 leader_election.cc:302] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9d41ff9d14f4bac8e76c97bcc15df96; no voters:
I20260923 17:19:44.864881 29914 leader_election.cc:288] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:44.865016 29919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:44.865253 29919 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [term 1 LEADER]: Becoming Leader. State: Replica: d9d41ff9d14f4bac8e76c97bcc15df96, State: Running, Role: LEADER
I20260923 17:19:44.865550 29919 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42687 } }
I20260923 17:19:44.866236 29914 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:44.867370 29920 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42687 } } }
I20260923 17:19:44.867362 29921 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9d41ff9d14f4bac8e76c97bcc15df96. Latest consensus state: current_term: 1 leader_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d41ff9d14f4bac8e76c97bcc15df96" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42687 } } }
I20260923 17:19:44.867527 29920 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:44.867537 29921 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:44.868048 29928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:44.870985 29928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:44.877336 29928 catalog_manager.cc:1347] Generated new cluster ID: a6aa1a81c54a4c52b67050cabf9fde77
I20260923 17:19:44.877450 29928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:44.896234 29928 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:44.897542 29928 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:44.905850 29928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9d41ff9d14f4bac8e76c97bcc15df96: Generated new TSK 0
I20260923 17:19:44.906632 29928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:19:45.037482 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:44.923481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.23.214.190:42687)
0923 17:19:44.923634 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:44.923641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:44.923815 (+ 174us) server_negotiation.cc:407] Connection header received
0923 17:19:44.923979 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:44.923982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:44.924031 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:44.924125 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:19:44.924392 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.924945 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.926229 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.926459 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.926547 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.926883 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:19:45.036853 (+109970us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:19:45.037264 (+ 411us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36618: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 17:19:45.037597 29917 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36618: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 17:19:45.037645 29943 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:44.923388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42687 (local address 127.0.0.1:36618)
0923 17:19:44.923715 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:44.923732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:44.923857 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:44.924152 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:44.924155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:44.924163 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:19:44.924285 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:44.924291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.925061 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.925065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:44.926100 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:44.926111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:44.926585 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:44.926589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:44.926677 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:44.926707 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:19:45.037161 (+110454us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:19:45.037411 (+ 250us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.214.190:42687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
I20260923 17:19:45.038421 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29847
2026-09-23T17:19:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (415 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (129 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:45 dist-test-slave-9q7f krb5kdc[29955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:45 dist-test-slave-9q7f krb5kdc[29955](info): set up 2 sockets
Sep 23 17:19:45 dist-test-slave-9q7f krb5kdc[29955](info): commencing operation
krb5kdc: starting...
W20260923 17:19:46.344379 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:46 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:46.477676 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35855
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:35855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:46.592248 29968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:46.592536 29968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:46.592578 29968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:46.596758 29968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:46.596820 29968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:46.596840 29968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:46.596856 29968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:46.596918 29968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:46.602795 29968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:35855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:46.603890 29968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:46.604941 29968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:46.613890 29974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:46.613955 29976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:46.613934 29973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:46.614362 29968 server_base.cc:1034] running on GCE node
I20260923 17:19:46.614928 29968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:46.616211 29968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:46.617393 29968 hybrid_clock.cc:648] HybridClock initialized: now 1790183986617362 us; error 45 us; skew 500 ppm
Sep 23 17:19:46 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183986, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:46.629187 29968 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:19:46.630378 29968 webserver.cc:460] Webserver started at http://127.23.214.190:37169/ using document root <none> and password file <none>
I20260923 17:19:46.630970 29968 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:46.631037 29968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:46.631217 29968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:46.633426 29968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "69c31c2228e7419cb0c128a4c6482482"
format_stamp: "Formatted at 2026-09-23 17:19:46 on dist-test-slave-9q7f"
server_key: "2a517e824b0ef0885f5013fc80746dd7"
server_key_iv: "2f33a8266c4c96bc3e16ce9f6e1eaa09"
server_key_version: "encryptionkey@0"
I20260923 17:19:46.634084 29968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "69c31c2228e7419cb0c128a4c6482482"
format_stamp: "Formatted at 2026-09-23 17:19:46 on dist-test-slave-9q7f"
server_key: "2a517e824b0ef0885f5013fc80746dd7"
server_key_iv: "2f33a8266c4c96bc3e16ce9f6e1eaa09"
server_key_version: "encryptionkey@0"
I20260923 17:19:46.638206 29968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:19:46.640786 29983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:46.642185 29968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:46.642334 29968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "69c31c2228e7419cb0c128a4c6482482"
format_stamp: "Formatted at 2026-09-23 17:19:46 on dist-test-slave-9q7f"
server_key: "2a517e824b0ef0885f5013fc80746dd7"
server_key_iv: "2f33a8266c4c96bc3e16ce9f6e1eaa09"
server_key_version: "encryptionkey@0"
I20260923 17:19:46.642465 29968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:46.671430 29968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:46.674788 29968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:46.675022 29968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:46.682745 29968 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:35855
I20260923 17:19:46.682770 30035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:35855 every 8 connection(s)
I20260923 17:19:46.683807 29968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:46.684664 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 29968
I20260923 17:19:46.684790 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:46.685077 24410 external_mini_cluster.cc:1428] Setting key 007b54a86124daa2757a39d6aa5e47fd
I20260923 17:19:46.687196 30036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:46.694173 30036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482: Bootstrap starting.
Sep 23 17:19:46 dist-test-slave-9q7f krb5kdc[29955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:46.697041 30036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:46.698011 30036 log.cc:824] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:46.700417 30036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482: No bootstrap required, opened a new log
I20260923 17:19:46.701190 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:46.686617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.23.214.190:35855)
0923 17:19:46.687066 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:46.687087 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:46.687129 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:19:46.687980 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:46.688004 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:46.688423 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:46.688801 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:46.689645 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:46.691193 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:46.692091 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:46.692787 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:46.692975 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:46.695877 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:46.695883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:46.695899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:46.695931 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:46.698050 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:46.698536 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:46.698540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:46.698545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:46.698661 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:46.698980 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:46.698983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:46.698984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:46.699464 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:46.699651 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:46.699891 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 17:19:46.700182 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20260923 17:19:46.704465 30036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c31c2228e7419cb0c128a4c6482482" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35855 } }
I20260923 17:19:46.704686 30036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:46.704730 30036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c31c2228e7419cb0c128a4c6482482, State: Initialized, Role: FOLLOWER
I20260923 17:19:46.705091 30036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c31c2228e7419cb0c128a4c6482482" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35855 } }
I20260923 17:19:46.705200 30036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:46.705238 30036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:46.705307 30036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:46.706385 30036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c31c2228e7419cb0c128a4c6482482" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35855 } }
I20260923 17:19:46.706748 30036 leader_election.cc:302] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c31c2228e7419cb0c128a4c6482482; no voters:
I20260923 17:19:46.707082 30036 leader_election.cc:288] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:46.707206 30041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:46.707479 30041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [term 1 LEADER]: Becoming Leader. State: Replica: 69c31c2228e7419cb0c128a4c6482482, State: Running, Role: LEADER
I20260923 17:19:46.707785 30041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c31c2228e7419cb0c128a4c6482482" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35855 } }
I20260923 17:19:46.708289 30036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:46.709432 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c31c2228e7419cb0c128a4c6482482. Latest consensus state: current_term: 1 leader_uuid: "69c31c2228e7419cb0c128a4c6482482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c31c2228e7419cb0c128a4c6482482" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35855 } } }
I20260923 17:19:46.709568 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:46.709830 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c31c2228e7419cb0c128a4c6482482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c31c2228e7419cb0c128a4c6482482" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35855 } } }
I20260923 17:19:46.709967 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:46.710060 30050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:46.713266 30050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:46.719019 30050 catalog_manager.cc:1347] Generated new cluster ID: 6ea5a546691d4e73a3631c4567fdad19
I20260923 17:19:46.719094 30050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:46.727788 30050 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:46.728729 30050 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:46.734864 30050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c31c2228e7419cb0c128a4c6482482: Generated new TSK 0
I20260923 17:19:46.735548 30050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:19:46.769659 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35855
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:46.886647 30062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:46.886931 30062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:46.886976 30062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:46.891053 30062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:46.891120 30062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:46.891181 30062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:46.896709 30062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:46.897800 30062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:46.898828 30062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:46.909441 30067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:46.909400 30070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:46.909680 30062 server_base.cc:1034] running on GCE node
W20260923 17:19:46.909411 30068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:46.910348 30062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:46.911129 30062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:46.912406 30062 hybrid_clock.cc:648] HybridClock initialized: now 1790183986912383 us; error 35 us; skew 500 ppm
Sep 23 17:19:46 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183986, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:46.915759 30062 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:19:46.916738 30062 webserver.cc:460] Webserver started at http://127.23.214.129:35469/ using document root <none> and password file <none>
I20260923 17:19:46.917357 30062 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:46.917461 30062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:46.917774 30062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:46.919956 30062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "6ecd6df681ff4cfabbad32a3fd5deccc"
format_stamp: "Formatted at 2026-09-23 17:19:46 on dist-test-slave-9q7f"
server_key: "3d911750e5780085af5aebfa23ae3848"
server_key_iv: "5a1e1b9ea34c4fb5abab178e26828748"
server_key_version: "encryptionkey@0"
I20260923 17:19:46.920568 30062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "6ecd6df681ff4cfabbad32a3fd5deccc"
format_stamp: "Formatted at 2026-09-23 17:19:46 on dist-test-slave-9q7f"
server_key: "3d911750e5780085af5aebfa23ae3848"
server_key_iv: "5a1e1b9ea34c4fb5abab178e26828748"
server_key_version: "encryptionkey@0"
I20260923 17:19:46.924710 30062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:19:46.927567 30077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:46.929070 30062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:46.929265 30062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "6ecd6df681ff4cfabbad32a3fd5deccc"
format_stamp: "Formatted at 2026-09-23 17:19:46 on dist-test-slave-9q7f"
server_key: "3d911750e5780085af5aebfa23ae3848"
server_key_iv: "5a1e1b9ea34c4fb5abab178e26828748"
server_key_version: "encryptionkey@0"
I20260923 17:19:46.929404 30062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:46.948917 30062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:46.953094 30062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:46.953329 30062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:46.954084 30062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:46.955047 30062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:46.955121 30062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:46.955222 30062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:46.955278 30062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:46.964581 30062 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:34571
I20260923 17:19:46.964653 30190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:34571 every 8 connection(s)
I20260923 17:19:46.965605 30062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:46.966601 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30062
I20260923 17:19:46.966742 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:46.967020 24410 external_mini_cluster.cc:1428] Setting key 17bb3d7acf522aaf8570c1d009841262
Sep 23 17:19:46 dist-test-slave-9q7f krb5kdc[29955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183986, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:46.982354 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:46.967753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:38661 (local address 127.23.214.190:35855)
0923 17:19:46.967887 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:46.967890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:46.968980 (+ 1090us) server_negotiation.cc:407] Connection header received
0923 17:19:46.970793 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:46.970796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:46.970849 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:46.970930 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:46.972369 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:46.972932 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:46.974986 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:46.975233 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:46.975332 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:46.978455 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:46.978462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:46.978465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:46.978500 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:46.980032 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:46.980600 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:46.980604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:46.980606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:46.980692 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:46.981069 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:46.981072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:46.981074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:46.981334 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:46.981443 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:46.982033 (+ 590us) server_negotiation.cc:299] Negotiation successful
0923 17:19:46.982178 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:19:46.983245 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:46.968244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35855 (local address 127.23.214.129:38661)
0923 17:19:46.968805 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:46.968864 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:46.970464 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:46.971173 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:46.971188 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:46.971678 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:46.972205 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:46.972219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:46.973078 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:46.973085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:46.974815 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:46.974832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:46.975426 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:46.975432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:46.975702 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:46.976403 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:46.976429 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:46.978231 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:46.980173 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:46.980179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:46.980202 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:46.980482 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:46.980791 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:46.980795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:46.980797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:46.980960 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:46.981443 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:46.981452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:46.981734 (+ 282us) client_negotiation.cc:769] Sending connection context
0923 17:19:46.981979 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 17:19:46.982373 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":152,"threads_started":1}
I20260923 17:19:46.984575 30191 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35855
I20260923 17:19:46.984853 30191 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:46.985389 30191 heartbeater.cc:511] Master 127.23.214.190:35855 requested a full tablet report, sending...
I20260923 17:19:46.987255 30000 ts_manager.cc:194] Registered new tserver with Master: 6ecd6df681ff4cfabbad32a3fd5deccc (127.23.214.129:34571)
I20260923 17:19:46.988610 30000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:38661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:19:46.999214 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35855
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:47.115096 30196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:47.115413 30196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:47.115496 30196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:47.119791 30196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:47.119886 30196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:47.119975 30196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:47.125547 30196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:47.126861 30196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:47.127904 30196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:47.138087 30202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:47.138087 30204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:47.138087 30201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:47.138207 30196 server_base.cc:1034] running on GCE node
I20260923 17:19:47.138868 30196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:47.139569 30196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:47.140761 30196 hybrid_clock.cc:648] HybridClock initialized: now 1790183987140720 us; error 50 us; skew 500 ppm
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.143996 30196 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:19:47.144994 30196 webserver.cc:460] Webserver started at http://127.23.214.130:46497/ using document root <none> and password file <none>
I20260923 17:19:47.145710 30196 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:47.145805 30196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:47.146060 30196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:47.148247 30196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "732b95714ed540c28b1afadf168456a1"
format_stamp: "Formatted at 2026-09-23 17:19:47 on dist-test-slave-9q7f"
server_key: "cf4ee16f5661c28bd3e5bf6477d5a715"
server_key_iv: "c01ae9801529f7ad148f81833373d1ed"
server_key_version: "encryptionkey@0"
I20260923 17:19:47.148823 30196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "732b95714ed540c28b1afadf168456a1"
format_stamp: "Formatted at 2026-09-23 17:19:47 on dist-test-slave-9q7f"
server_key: "cf4ee16f5661c28bd3e5bf6477d5a715"
server_key_iv: "c01ae9801529f7ad148f81833373d1ed"
server_key_version: "encryptionkey@0"
I20260923 17:19:47.152878 30196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:19:47.155658 30211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:47.156994 30196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:47.157156 30196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "732b95714ed540c28b1afadf168456a1"
format_stamp: "Formatted at 2026-09-23 17:19:47 on dist-test-slave-9q7f"
server_key: "cf4ee16f5661c28bd3e5bf6477d5a715"
server_key_iv: "c01ae9801529f7ad148f81833373d1ed"
server_key_version: "encryptionkey@0"
I20260923 17:19:47.157290 30196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:47.177008 30196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:47.180131 30196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:47.180357 30196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:47.180975 30196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:47.181836 30196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:47.181936 30196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:47.182025 30196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:47.182078 30196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:47.191638 30196 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:38125
I20260923 17:19:47.191680 30324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:38125 every 8 connection(s)
I20260923 17:19:47.192593 30196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:47.196239 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30196
I20260923 17:19:47.196355 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:47.196635 24410 external_mini_cluster.cc:1428] Setting key e564cb457c4be8a1f9cf954e5dff8d3f
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:19:47.208016 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:47.194526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:45375 (local address 127.23.214.190:35855)
0923 17:19:47.194682 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.194687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.195660 (+ 973us) server_negotiation.cc:407] Connection header received
0923 17:19:47.197151 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.197157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.197225 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.197306 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.198749 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.199274 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.200420 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.200642 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.200752 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:47.203577 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:47.203583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:47.203590 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:47.203627 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:47.205580 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.206238 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.206241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.206243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.206350 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.206782 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.206788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.206791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.207049 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:47.207149 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:47.207768 (+ 619us) server_negotiation.cc:299] Negotiation successful
0923 17:19:47.207887 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:19:47.209228 30327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:47.194970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35855 (local address 127.23.214.130:45375)
0923 17:19:47.195502 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:47.195546 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:47.196928 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:47.197614 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:47.197621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:47.198080 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:47.198604 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:47.198618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.199399 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.199403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:47.200295 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:47.200304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.200798 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.200802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:47.200971 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:47.201571 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:47.201596 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:47.203387 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:47.205750 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:47.205755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:47.205770 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:47.206132 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:47.206489 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:47.206492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:47.206493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:47.206646 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:47.207162 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:47.207168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:47.207494 (+ 326us) client_negotiation.cc:769] Sending connection context
0923 17:19:47.207667 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 17:19:47.207986 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
I20260923 17:19:47.210829 30325 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35855
I20260923 17:19:47.211103 30325 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:47.211676 30325 heartbeater.cc:511] Master 127.23.214.190:35855 requested a full tablet report, sending...
I20260923 17:19:47.212841 30000 ts_manager.cc:194] Registered new tserver with Master: 732b95714ed540c28b1afadf168456a1 (127.23.214.130:38125)
I20260923 17:19:47.213454 30000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:45375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:19:47.228952 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35855
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:47.343276 30330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:47.343540 30330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:47.343617 30330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:47.347931 30330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:47.347996 30330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:47.348104 30330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:47.353554 30330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:35855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:47.354738 30330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:47.355777 30330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:47.365752 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:47.365800 30338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:47.365801 30335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:47.366158 30330 server_base.cc:1034] running on GCE node
I20260923 17:19:47.366626 30330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:47.367302 30330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:47.368496 30330 hybrid_clock.cc:648] HybridClock initialized: now 1790183987368479 us; error 30 us; skew 500 ppm
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.371382 30330 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:19:47.372339 30330 webserver.cc:460] Webserver started at http://127.23.214.131:35893/ using document root <none> and password file <none>
I20260923 17:19:47.372920 30330 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:47.373011 30330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:47.373250 30330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:47.375380 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "b81396e5ab0c43e798f0801a00e497e3"
format_stamp: "Formatted at 2026-09-23 17:19:47 on dist-test-slave-9q7f"
server_key: "7ae807bb848d169139c71742be848b29"
server_key_iv: "627a1fd07677ae3fce2ecdd6220e9243"
server_key_version: "encryptionkey@0"
I20260923 17:19:47.375945 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "b81396e5ab0c43e798f0801a00e497e3"
format_stamp: "Formatted at 2026-09-23 17:19:47 on dist-test-slave-9q7f"
server_key: "7ae807bb848d169139c71742be848b29"
server_key_iv: "627a1fd07677ae3fce2ecdd6220e9243"
server_key_version: "encryptionkey@0"
I20260923 17:19:47.379676 30330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:19:47.382241 30345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:47.383505 30330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:19:47.383642 30330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "b81396e5ab0c43e798f0801a00e497e3"
format_stamp: "Formatted at 2026-09-23 17:19:47 on dist-test-slave-9q7f"
server_key: "7ae807bb848d169139c71742be848b29"
server_key_iv: "627a1fd07677ae3fce2ecdd6220e9243"
server_key_version: "encryptionkey@0"
I20260923 17:19:47.383760 30330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:47.406890 30330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:47.410794 30330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:47.411012 30330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:47.411685 30330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:47.412575 30330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:47.412648 30330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:47.412726 30330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:47.412778 30330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:47.422238 30330 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:37445
I20260923 17:19:47.422264 30458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:37445 every 8 connection(s)
I20260923 17:19:47.423261 30330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:47.426084 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30330
I20260923 17:19:47.426200 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:47.426585 24410 external_mini_cluster.cc:1428] Setting key 50c22d91aea73cbb13ed3d6894aea103
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:19:47.438129 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:47.425194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:33499 (local address 127.23.214.190:35855)
0923 17:19:47.425387 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.425391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.426370 (+ 979us) server_negotiation.cc:407] Connection header received
0923 17:19:47.428154 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.428157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.428221 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.428321 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.429573 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.430203 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.431404 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.431633 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.431748 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:47.434552 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:47.434555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:47.434557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:47.434586 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:47.435922 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.436549 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.436552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.436553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.436634 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.437004 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.437006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.437007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.437228 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:47.437316 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:47.437820 (+ 504us) server_negotiation.cc:299] Negotiation successful
0923 17:19:47.438000 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 17:19:47.439056 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:47.425617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:35855 (local address 127.23.214.131:33499)
0923 17:19:47.426191 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:47.426255 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:47.427932 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:47.428525 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:47.428533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:47.428914 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:47.429426 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:47.429441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.430347 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.430351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:47.431266 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:47.431277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.431801 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.431805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:47.431980 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:47.432572 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:47.432589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:47.434335 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:47.436053 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:47.436059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:47.436073 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:47.436434 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:47.436744 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:47.436747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:47.436749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:47.436911 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:47.437323 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:47.437328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:47.437589 (+ 261us) client_negotiation.cc:769] Sending connection context
0923 17:19:47.437760 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 17:19:47.438107 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":127,"threads_started":1}
I20260923 17:19:47.440361 30459 heartbeater.cc:347] Connected to a master server at 127.23.214.190:35855
I20260923 17:19:47.440639 30459 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:47.441190 30459 heartbeater.cc:511] Master 127.23.214.190:35855 requested a full tablet report, sending...
I20260923 17:19:47.442327 29999 ts_manager.cc:194] Registered new tserver with Master: b81396e5ab0c43e798f0801a00e497e3 (127.23.214.131:37445)
I20260923 17:19:47.443198 29999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:33499
I20260923 17:19:47.453209 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/test-admin.keytab.
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.471724 24410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 17:19:47.474584 30466 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:47.473141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.23.214.129:34571)
0923 17:19:47.473458 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.473473 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.473499 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:19:47.473549 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.473554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.473746 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.473929 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.474414 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56848: BlockingRecv error: recv got EOF from 127.0.0.1:56848 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":117,"threads_started":1}
W20260923 17:19:47.474679 24410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.475844 24410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.476889 24410 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:47.487567 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:47.477937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.23.214.129:34571)
0923 17:19:47.478082 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.478088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.478183 (+ 95us) server_negotiation.cc:407] Connection header received
0923 17:19:47.478352 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.478360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.478403 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.478488 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.479044 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.479874 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.480701 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.481101 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.481214 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:47.483754 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:47.483769 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:47.483788 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:47.483817 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:47.485626 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.486182 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.486188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.486189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.486272 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.486535 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.486537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.486538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.486780 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:47.486943 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:47.487179 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 17:19:47.487406 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 17:19:47.491840 30466 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:47.490644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.23.214.129:34571)
0923 17:19:47.490813 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.490817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.490959 (+ 142us) server_negotiation.cc:407] Connection header received
0923 17:19:47.491120 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.491123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.491172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.491265 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.491740 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56866: BlockingRecv error: recv got EOF from 127.0.0.1:56866 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260923 17:19:47.492172 24410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.493216 24410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.494153 24410 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:47.503180 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:47.494934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.23.214.129:34571)
0923 17:19:47.495058 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.495062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.495190 (+ 128us) server_negotiation.cc:407] Connection header received
0923 17:19:47.495289 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.495291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.495329 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.495396 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.495868 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.496372 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.497207 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.497438 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.497527 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:47.500002 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:47.500006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:47.500008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:47.500034 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:47.501366 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.501787 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.501791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.501792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.501964 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.502256 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.502258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.502259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.502445 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:47.502553 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:47.502796 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 17:19:47.502964 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 17:19:47.506412 30466 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:47.505143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.23.214.129:34571)
0923 17:19:47.505283 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.505287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.505404 (+ 117us) server_negotiation.cc:407] Connection header received
0923 17:19:47.505562 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.505565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.505609 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.505703 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.506288 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56888: BlockingRecv error: recv got EOF from 127.0.0.1:56888 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 17:19:47.506482 24410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790183952943234-24410-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.507546 24410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:47.508460 24410 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[29955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:19:47.517591 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:47.509390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.23.214.129:34571)
0923 17:19:47.509529 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:47.509533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:47.509734 (+ 201us) server_negotiation.cc:407] Connection header received
0923 17:19:47.509800 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:47.509802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:47.509845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:47.510004 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:47.510515 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.511015 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.511907 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:47.512117 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:47.512197 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:47.514577 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:47.514581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:47.514583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:47.514607 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:47.515953 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.516382 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.516385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.516386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.516467 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:47.516725 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:47.516727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:47.516728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:47.516905 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:47.517021 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:47.517265 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:19:47.517439 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:19:47.518697 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30062
I20260923 17:19:47.524825 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30196
I20260923 17:19:47.530380 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30330
I20260923 17:19:47.535771 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 29968
2026-09-23T17:19:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2242 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[30479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[30479](info): set up 2 sockets
Sep 23 17:19:47 dist-test-slave-9q7f krb5kdc[30479](info): commencing operation
krb5kdc: starting...
W20260923 17:19:48.572623 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:48 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab.
Entry for principal oryx/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab.
I20260923 17:19:48.702142 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40645
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab
--principal=oryx/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:48.828738 30492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:48.829104 30492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:48.829190 30492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:48.833518 30492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:48.833645 30492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:48.833683 30492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:48.833724 30492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:48.833762 30492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:48.839465 30492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:40645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab
--principal=oryx/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:48.840673 30492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:48.841790 30492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:48.850672 30497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:48.850837 30498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:48.850672 30500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:48.851055 30492 server_base.cc:1034] running on GCE node
I20260923 17:19:48.851639 30492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:48.852865 30492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:48.854058 30492 hybrid_clock.cc:648] HybridClock initialized: now 1790183988854043 us; error 30 us; skew 500 ppm
Sep 23 17:19:48 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183988, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:48.857056 30492 init.cc:377] Logged in from keytab as oryx/127.23.214.190@KRBTEST.COM (short username oryx)
I20260923 17:19:48.858059 30492 webserver.cc:460] Webserver started at http://127.23.214.190:46093/ using document root <none> and password file <none>
I20260923 17:19:48.858608 30492 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:48.858672 30492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:48.858852 30492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:48.860937 30492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "e2697f8ca5a64bc591679a6e45f99c91"
format_stamp: "Formatted at 2026-09-23 17:19:48 on dist-test-slave-9q7f"
server_key: "4e61fdaad24cf7b44a5bae8014acf6a5"
server_key_iv: "ab0eb3d12c174db5328201e6916e53f4"
server_key_version: "encryptionkey@0"
I20260923 17:19:48.861500 30492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "e2697f8ca5a64bc591679a6e45f99c91"
format_stamp: "Formatted at 2026-09-23 17:19:48 on dist-test-slave-9q7f"
server_key: "4e61fdaad24cf7b44a5bae8014acf6a5"
server_key_iv: "ab0eb3d12c174db5328201e6916e53f4"
server_key_version: "encryptionkey@0"
I20260923 17:19:48.865417 30492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 17:19:48.868155 30507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:48.869520 30492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:48.869633 30492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "e2697f8ca5a64bc591679a6e45f99c91"
format_stamp: "Formatted at 2026-09-23 17:19:48 on dist-test-slave-9q7f"
server_key: "4e61fdaad24cf7b44a5bae8014acf6a5"
server_key_iv: "ab0eb3d12c174db5328201e6916e53f4"
server_key_version: "encryptionkey@0"
I20260923 17:19:48.869730 30492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:48.891773 30492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:48.895408 30492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:48.895584 30492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:48.903301 30492 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:40645
I20260923 17:19:48.903609 30559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:40645 every 8 connection(s)
I20260923 17:19:48.904384 30492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:48.907445 30560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:48.908963 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30492
I20260923 17:19:48.909132 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:48.909494 24410 external_mini_cluster.cc:1428] Setting key 644bd780f866dd9e607184aa3e86dc8f
I20260923 17:19:48.914069 30560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: Bootstrap starting.
I20260923 17:19:48.916447 30560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:48.917227 30560 log.cc:824] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:19:48 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:48.919723 30560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: No bootstrap required, opened a new log
I20260923 17:19:48.923607 30560 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } }
I20260923 17:19:48.923980 30560 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:48.924068 30560 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2697f8ca5a64bc591679a6e45f99c91, State: Initialized, Role: FOLLOWER
I20260923 17:19:48.924561 30560 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } }
I20260923 17:19:48.924719 30560 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:48.924808 30560 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:48.924906 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:48.911029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.23.214.190:40645)
0923 17:19:48.911519 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:48.911535 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:48.911573 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:19:48.912216 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:48.912234 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:48.912589 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:48.912948 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:48.913796 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:48.914914 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:48.915779 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:48.916276 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:48.916453 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:48.919439 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:48.919445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:48.919460 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:48.919491 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:48.921429 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:48.922009 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:48.922014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:48.922021 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:48.922147 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:48.922527 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:48.922531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:48.922533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:48.922999 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:48.923244 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:48.923528 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 17:19:48.923886 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":150,"threads_started":1}
I20260923 17:19:48.924926 30560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:48.926087 30560 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } }
I20260923 17:19:48.926457 30560 leader_election.cc:302] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2697f8ca5a64bc591679a6e45f99c91; no voters:
I20260923 17:19:48.926815 30560 leader_election.cc:288] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:48.926913 30565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:48.927093 30565 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 1 LEADER]: Becoming Leader. State: Replica: e2697f8ca5a64bc591679a6e45f99c91, State: Running, Role: LEADER
I20260923 17:19:48.927413 30565 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } }
I20260923 17:19:48.928220 30560 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:48.928759 30566 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2697f8ca5a64bc591679a6e45f99c91. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } }
I20260923 17:19:48.928930 30566 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:48.929279 30567 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } }
I20260923 17:19:48.929399 30567 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:48.930334 30570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:48.933365 30570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:48.939751 30570 catalog_manager.cc:1347] Generated new cluster ID: 3ad1c6fb21b34c86a4afaefcd9e7447d
I20260923 17:19:48.939851 30570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:48.959517 30570 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:48.960880 30570 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:48.973682 30570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: Generated new TSK 0
I20260923 17:19:48.974563 30570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab.
Entry for principal oryx/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab.
I20260923 17:19:49.018077 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40645
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab
--principal=oryx/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:49.139323 30586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:49.139633 30586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:49.139679 30586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:49.143972 30586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:49.144053 30586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:49.144116 30586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:49.150177 30586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab
--principal=oryx/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:49.151387 30586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:49.152487 30586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:49.163216 30594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:49.163298 30591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:49.163335 30586 server_base.cc:1034] running on GCE node
W20260923 17:19:49.163353 30592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:49.163978 30586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:49.164930 30586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:49.168263 30586 hybrid_clock.cc:648] HybridClock initialized: now 1790183989168221 us; error 45 us; skew 500 ppm
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:49.171779 30586 init.cc:377] Logged in from keytab as oryx/127.23.214.129@KRBTEST.COM (short username oryx)
I20260923 17:19:49.172850 30586 webserver.cc:460] Webserver started at http://127.23.214.129:34489/ using document root <none> and password file <none>
I20260923 17:19:49.173674 30586 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:49.173753 30586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:49.174031 30586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:49.176494 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "cb655b09bcae49eaaf4c91acf1506549"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "4d29ef533aa89681cb5d59d3e0546af3"
server_key_iv: "96b1b0be0ce7449d5518caba67420b11"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.177151 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "cb655b09bcae49eaaf4c91acf1506549"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "4d29ef533aa89681cb5d59d3e0546af3"
server_key_iv: "96b1b0be0ce7449d5518caba67420b11"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.181469 30586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:19:49.184288 30601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.185779 30586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:49.185967 30586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "cb655b09bcae49eaaf4c91acf1506549"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "4d29ef533aa89681cb5d59d3e0546af3"
server_key_iv: "96b1b0be0ce7449d5518caba67420b11"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.186120 30586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:49.206215 30586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:49.210147 30586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:49.210394 30586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:49.211098 30586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:49.212070 30586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:49.212150 30586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.212242 30586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:49.212306 30586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.222301 30586 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:34811
I20260923 17:19:49.222312 30714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:34811 every 8 connection(s)
I20260923 17:19:49.223399 30586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:49.225844 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30586
I20260923 17:19:49.226042 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:49.226414 24410 external_mini_cluster.cc:1428] Setting key 6703c5791082bcabe17773f9ca7e40d9
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.130@KRBTEST.COM" created.
I20260923 17:19:49.241057 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.225349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:47063 (local address 127.23.214.190:40645)
0923 17:19:49.225504 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:49.225516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:49.226486 (+ 970us) server_negotiation.cc:407] Connection header received
0923 17:19:49.228430 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:49.228433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:49.228476 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:49.228536 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:49.230478 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.231146 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.233189 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.233448 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.233544 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.237041 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:49.237044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:49.237046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:49.237073 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:49.238665 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.239284 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.239287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.239289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.239386 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.239726 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.239729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.239730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.239975 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:49.240069 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:49.240787 (+ 718us) server_negotiation.cc:299] Negotiation successful
0923 17:19:49.240926 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:19:49.242224 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.225682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:47063)
0923 17:19:49.226263 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:49.226356 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:49.228166 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:49.228740 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:49.228754 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:49.229394 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:49.230298 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.230318 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.231307 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.231315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.233040 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.233057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.233641 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.233647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.233952 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.234686 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:49.234717 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:49.236815 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:49.238793 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.238800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.238817 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.239173 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.239478 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.239481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.239482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.239635 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.240113 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:49.240121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:49.240417 (+ 296us) client_negotiation.cc:769] Sending connection context
0923 17:19:49.240717 (+ 300us) client_negotiation.cc:240] Negotiation successful
0923 17:19:49.241173 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":184,"threads_started":1}
I20260923 17:19:49.243767 30715 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40645
I20260923 17:19:49.244031 30715 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:49.244565 30715 heartbeater.cc:511] Master 127.23.214.190:40645 requested a full tablet report, sending...
I20260923 17:19:49.246322 30524 ts_manager.cc:194] Registered new tserver with Master: cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811)
I20260923 17:19:49.247481 30524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:47063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab.
Entry for principal oryx/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab.
I20260923 17:19:49.258320 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40645
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab
--principal=oryx/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:49.376236 30720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:49.376556 30720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:49.376603 30720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:49.380746 30720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:49.380823 30720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:49.380885 30720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:49.386847 30720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab
--principal=oryx/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:49.388532 30720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:49.390453 30720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:49.405021 30726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:49.405037 30728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:49.405038 30725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:49.405274 30720 server_base.cc:1034] running on GCE node
I20260923 17:19:49.405773 30720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:49.406607 30720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:49.407820 30720 hybrid_clock.cc:648] HybridClock initialized: now 1790183989407799 us; error 36 us; skew 500 ppm
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:49.410972 30720 init.cc:377] Logged in from keytab as oryx/127.23.214.130@KRBTEST.COM (short username oryx)
I20260923 17:19:49.411959 30720 webserver.cc:460] Webserver started at http://127.23.214.130:37947/ using document root <none> and password file <none>
I20260923 17:19:49.412596 30720 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:49.412694 30720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:49.412945 30720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:49.415244 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "b5b3cb4911834d8287091fdbdac42d2a"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "a301525eb7d2d30ff2a18fbcc4e66e0c"
server_key_iv: "6554709c7911af74f03ab35bbe951259"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.415870 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "b5b3cb4911834d8287091fdbdac42d2a"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "a301525eb7d2d30ff2a18fbcc4e66e0c"
server_key_iv: "6554709c7911af74f03ab35bbe951259"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.419924 30720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:49.422705 30735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.424234 30720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:19:49.424388 30720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "b5b3cb4911834d8287091fdbdac42d2a"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "a301525eb7d2d30ff2a18fbcc4e66e0c"
server_key_iv: "6554709c7911af74f03ab35bbe951259"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.424521 30720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:49.452376 30720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:49.456290 30720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:49.456555 30720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:49.457199 30720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:49.458163 30720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:49.458233 30720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.458318 30720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:49.458366 30720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.468115 30720 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:35989
I20260923 17:19:49.468163 30848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:35989 every 8 connection(s)
I20260923 17:19:49.469102 30720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:49.475769 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30720
I20260923 17:19:49.475880 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:49.476200 24410 external_mini_cluster.cc:1428] Setting key 892b78749df8f925d88ba596eecc4426
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:49.485085 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.471148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:46647 (local address 127.23.214.190:40645)
0923 17:19:49.471286 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:49.471290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:49.472222 (+ 932us) server_negotiation.cc:407] Connection header received
0923 17:19:49.473833 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:49.473837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:49.473936 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:49.474045 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:49.475337 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.475957 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.477573 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.477849 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.478003 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.480954 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:49.480961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:49.480968 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:49.481003 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:49.482884 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.483482 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.483486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.483488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.483576 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.483935 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.483940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.483942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.484144 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:49.484246 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:49.484785 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 17:19:49.484918 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for oryx/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:19:49.486115 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.471600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:46647)
0923 17:19:49.472070 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:49.472115 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:49.473569 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:49.474238 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:49.474247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:49.474650 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:49.475159 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.475173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.476119 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.476123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.477463 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.477473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.478073 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.478077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.478331 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.479009 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:49.479036 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:49.480770 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:49.483050 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.483056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.483077 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.483364 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.483678 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.483681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.483683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.483831 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.484259 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:49.484267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:49.484538 (+ 271us) client_negotiation.cc:769] Sending connection context
0923 17:19:49.484748 (+ 210us) client_negotiation.cc:240] Negotiation successful
0923 17:19:49.485112 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20260923 17:19:49.487354 30849 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40645
I20260923 17:19:49.487619 30849 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:49.488155 30849 heartbeater.cc:511] Master 127.23.214.190:40645 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.131@KRBTEST.COM" created.
I20260923 17:19:49.489499 30524 ts_manager.cc:194] Registered new tserver with Master: b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
I20260923 17:19:49.490257 30524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:46647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab.
Entry for principal oryx/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab.
I20260923 17:19:49.505529 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40645
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab
--principal=oryx/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:49.625531 30854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:49.625865 30854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:49.625996 30854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:49.631155 30854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:49.631244 30854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:49.631318 30854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:49.637717 30854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab
--principal=oryx/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:49.639101 30854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:49.640290 30854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:49.651861 30860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:49.651932 30854 server_base.cc:1034] running on GCE node
W20260923 17:19:49.651883 30862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:49.651861 30859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:49.652827 30854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:49.653622 30854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:49.654840 30854 hybrid_clock.cc:648] HybridClock initialized: now 1790183989654802 us; error 52 us; skew 500 ppm
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:49.658008 30854 init.cc:377] Logged in from keytab as oryx/127.23.214.131@KRBTEST.COM (short username oryx)
I20260923 17:19:49.659191 30854 webserver.cc:460] Webserver started at http://127.23.214.131:42059/ using document root <none> and password file <none>
I20260923 17:19:49.659803 30854 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:49.659871 30854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:49.660063 30854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:49.662477 30854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "c37d82581dd14b6db7223f55d05e2da0"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "10779e60a7e9d74556a2f2d4a062096e"
server_key_iv: "d049f9c0e7e731958200eae058132849"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.663071 30854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "c37d82581dd14b6db7223f55d05e2da0"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "10779e60a7e9d74556a2f2d4a062096e"
server_key_iv: "d049f9c0e7e731958200eae058132849"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.667353 30854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:19:49.670143 30869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.671571 30854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:49.671743 30854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "c37d82581dd14b6db7223f55d05e2da0"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "10779e60a7e9d74556a2f2d4a062096e"
server_key_iv: "d049f9c0e7e731958200eae058132849"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.671890 30854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:49.704205 30854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:49.707769 30854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:49.708001 30854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:49.708642 30854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:49.709525 30854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:49.709596 30854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.709666 30854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:49.709719 30854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.718936 30854 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:41939
I20260923 17:19:49.718958 30982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:41939 every 8 connection(s)
I20260923 17:19:49.719900 30854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:49.723151 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30854
I20260923 17:19:49.723276 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:49.723578 24410 external_mini_cluster.cc:1428] Setting key 3a5db44a8dc3fd6f7c88d8fe8a482344
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:49.734404 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.721843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:32989 (local address 127.23.214.190:40645)
0923 17:19:49.722012 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:49.722015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:49.722897 (+ 882us) server_negotiation.cc:407] Connection header received
0923 17:19:49.724647 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:49.724650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:49.724707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:49.724783 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:49.726143 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.726675 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.727861 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.728085 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.728216 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.730955 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:49.730959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:49.730961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:49.730986 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:49.732277 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.732810 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.732814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.732815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.732899 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.733249 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.733252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.733254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.733446 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:49.733548 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:49.734105 (+ 557us) server_negotiation.cc:299] Negotiation successful
0923 17:19:49.734234 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:19:49.735271 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.722259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:32989)
0923 17:19:49.722720 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:49.722765 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:49.724437 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:49.724990 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:49.725002 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:49.725400 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:49.726004 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.726018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.726798 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.726802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.727726 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.727735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.728314 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.728319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.728484 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.729128 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:49.729145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:49.730764 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:49.732401 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.732408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.732425 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.732692 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.732996 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.732999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.733001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.733146 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.733546 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:49.733554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:49.733818 (+ 264us) client_negotiation.cc:769] Sending connection context
0923 17:19:49.734054 (+ 236us) client_negotiation.cc:240] Negotiation successful
0923 17:19:49.734419 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260923 17:19:49.736505 30983 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40645
I20260923 17:19:49.736780 30983 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:49.737434 30983 heartbeater.cc:511] Master 127.23.214.190:40645 requested a full tablet report, sending...
I20260923 17:19:49.738593 30524 ts_manager.cc:194] Registered new tserver with Master: c37d82581dd14b6db7223f55d05e2da0 (127.23.214.131:41939)
I20260923 17:19:49.739171 30524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:32989
I20260923 17:19:49.750010 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.214.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.189.keytab.
Entry for principal oryx/127.23.214.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.189.keytab.
I20260923 17:19:49.780335 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:43711
--webserver_interface=127.23.214.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.189.keytab
--principal=oryx/127.23.214.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:40645,127.23.214.189:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:49.895087 30988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:49.895392 30988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:49.895434 30988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:49.899792 30988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:49.899865 30988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:49.899887 30988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:49.899904 30988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:49.899920 30988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:49.906549 30988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:40645,127.23.214.189:43711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.189.keytab
--principal=oryx/127.23.214.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.214.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
W20260923 17:19:49.906813 30988 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.214.190:40645,127.23.214.189:43711'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 17:19:49.907896 30988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:49.909214 30988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:49.919443 30996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:49.919451 30993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:49.919436 30994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:49.919761 30988 server_base.cc:1034] running on GCE node
I20260923 17:19:49.920400 30988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:49.921655 30988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:49.922864 30988 hybrid_clock.cc:648] HybridClock initialized: now 1790183989922842 us; error 64 us; skew 500 ppm
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:49.925918 30988 init.cc:377] Logged in from keytab as oryx/127.23.214.189@KRBTEST.COM (short username oryx)
I20260923 17:19:49.926934 30988 webserver.cc:460] Webserver started at http://127.23.214.189:33203/ using document root <none> and password file <none>
I20260923 17:19:49.927584 30988 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:49.927683 30988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:49.927987 30988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:49.930475 30988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/instance:
uuid: "65c45e932d664f29b5e591afdd6bd804"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "ead177582a915679c38934fd00c9be8f"
server_key_iv: "5d2c882b1db6241880f9eda40f2c31b5"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.931104 30988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal/instance:
uuid: "65c45e932d664f29b5e591afdd6bd804"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "ead177582a915679c38934fd00c9be8f"
server_key_iv: "5d2c882b1db6241880f9eda40f2c31b5"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.935068 30988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:19:49.937798 31003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:49.939194 30988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:49.939352 30988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
uuid: "65c45e932d664f29b5e591afdd6bd804"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "ead177582a915679c38934fd00c9be8f"
server_key_iv: "5d2c882b1db6241880f9eda40f2c31b5"
server_key_version: "encryptionkey@0"
I20260923 17:19:49.939488 30988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:49.958664 30988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:49.962088 30988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:49.962301 30988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:49.969621 30988 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.189:43711
I20260923 17:19:49.969983 31055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.189:43711 every 8 connection(s)
I20260923 17:19:49.970777 30988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
I20260923 17:19:49.973989 31056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:49.977252 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 30988
I20260923 17:19:49.978780 31056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.189@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.214.189@KRBTEST.COM
I20260923 17:19:49.991377 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.980708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.23.214.190:40645)
0923 17:19:49.980860 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:49.980864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:49.981247 (+ 383us) server_negotiation.cc:407] Connection header received
0923 17:19:49.981486 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:49.981489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:49.981541 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:49.981629 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:49.982552 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.983089 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.984288 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.984548 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.984655 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.987319 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:49.987322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:49.987325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:49.987351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:49.988830 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.989387 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.989391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.989393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.989478 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.989998 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.990000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.990002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.990348 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:49.990472 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:49.991070 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 17:19:49.991200 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:19:49.992285 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.978998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.23.214.189:43711)
0923 17:19:49.979467 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:49.979483 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:49.979521 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:19:49.980162 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:49.980179 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:49.980572 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:49.980959 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:49.981763 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.983016 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.983925 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.984417 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.984531 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.987771 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:49.987780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:49.987796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:49.987828 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:49.989584 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.990126 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.990132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.990134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.990216 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:49.990574 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:49.990578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:49.990580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:49.990837 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:49.991059 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:49.991262 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 17:19:49.991409 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"spinlock_wait_cycles":896,"thread_start_us":136,"threads_started":1}
I20260923 17:19:49.992290 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.980640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38220)
0923 17:19:49.981123 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:49.981172 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:49.981326 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:49.981683 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:49.981689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:49.982174 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:49.982380 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.982394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.983230 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.983232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.984130 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.984138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.984711 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.984715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.984795 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.985520 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:49.985536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:49.987147 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:49.988959 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.988969 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.988985 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.989271 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.989573 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:49.989578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:49.989580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:49.989797 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:49.990475 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:49.990485 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:49.990768 (+ 283us) client_negotiation.cc:769] Sending connection context
0923 17:19:49.990958 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 17:19:49.991335 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1,"thread_start_us":198,"threads_started":1}
I20260923 17:19:49.993682 31056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:19:49 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183989, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.189@KRBTEST.COM for oryx/127.23.214.189@KRBTEST.COM
I20260923 17:19:50.002048 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.994225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.0.0.1:56542)
0923 17:19:49.994353 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:49.994370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:49.994462 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:49.994705 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:49.994707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:49.994888 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:49.994981 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.994986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.996031 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.996034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.996612 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:49.996621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.997152 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.997155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:49.997281 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.997757 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:49.997768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:49.999166 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.000701 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.000704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.000706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.000910 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.001166 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.001168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.001170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.001243 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.001592 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.001594 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.001667 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.001744 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.001842 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:19:50.002066 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:49.994302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.23.214.189:43711)
0923 17:19:49.994402 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:49.994407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:49.994422 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:19:49.994548 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:49.994551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:49.994599 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:49.994660 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:49.995075 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.995909 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.996743 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:49.997000 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:49.997096 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:49.999294 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:49.999298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:49.999301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:49.999324 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.000593 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.000989 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.000991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.000993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.001058 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.001327 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.001330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.001332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.001512 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.001588 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.001813 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.001960 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:19:50.004918 31056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Bootstrap starting.
I20260923 17:19:50.007357 31056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:50.008131 31056 log.cc:824] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:50.010299 31056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: No bootstrap required, opened a new log
I20260923 17:19:50.013293 31056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } }
I20260923 17:19:50.013533 31056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:50.013615 31056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c45e932d664f29b5e591afdd6bd804, State: Initialized, Role: FOLLOWER
I20260923 17:19:50.014124 31056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } }
I20260923 17:19:50.014613 31062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } } }
I20260923 17:19:50.014724 31062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:19:50.015349 31056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:19:50.020215 31073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:19:50.020309 31073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:19:50.022399 30988 master_runner.cc:428] Detected that this master 65c45e932d664f29b5e591afdd6bd804 is joining an existing cluster
I20260923 17:19:50.022471 30988 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.214.189:43711
I20260923 17:19:50.034108 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.024939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38230)
0923 17:19:50.025776 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.025788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.025930 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.026173 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.026176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.026457 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.026555 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.026561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.027622 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.027625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.028604 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.028614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.029280 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.029293 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.029412 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.029936 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.029952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.030441 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.032482 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.032484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.032486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.032727 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.033030 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.033033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.033035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.033127 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.033525 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.033528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.033628 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.033735 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.033902 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":152,"thread_start_us":100,"threads_started":1}
I20260923 17:19:50.034163 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.025021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.23.214.190:40645)
0923 17:19:50.025281 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.025286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.025847 (+ 561us) server_negotiation.cc:407] Connection header received
0923 17:19:50.026023 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.026025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.026077 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.026161 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.026688 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.027503 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.028719 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.029054 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.029157 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.030585 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.030590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.030596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.030624 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.032357 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.032829 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.032833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.032835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.032931 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.033232 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.033238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.033240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.033435 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.033513 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.033818 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.033970 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260923 17:19:50.034637 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.025144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.23.214.189:43711)
0923 17:19:50.025313 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.025318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.025518 (+ 200us) server_negotiation.cc:407] Connection header received
0923 17:19:50.025662 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.025665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.025705 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.025781 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.026392 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.026998 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.027900 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.028364 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.028463 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.030451 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.030455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.030458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.030479 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.032296 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.032897 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.032901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.032902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.032972 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.033549 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.033552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.033553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.033695 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.033807 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.034118 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.034508 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 17:19:50.035099 31080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.024938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.0.0.1:56546)
0923 17:19:50.025434 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.025445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.025563 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.025839 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.025842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.026105 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.026257 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.026266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.027116 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.027119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.027782 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.027790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.028511 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.028515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.028654 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.029851 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.029870 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.030309 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.032417 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.032420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.032423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.032779 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.033065 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.033067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.033069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.033139 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.033841 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.033843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.033964 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.034808 (+ 844us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.034941 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":20,"thread_start_us":104,"threads_started":1}
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183988, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.190@KRBTEST.COM for oryx/127.23.214.189@KRBTEST.COM
I20260923 17:19:50.048981 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.038792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.23.214.189:43711)
0923 17:19:50.038949 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.038953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.039358 (+ 405us) server_negotiation.cc:407] Connection header received
0923 17:19:50.039539 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.039541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.039587 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.039652 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.040447 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.040917 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.042175 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.042451 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.042528 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.045241 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.045249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.045251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.045273 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.047021 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.047644 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.047647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.047649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.047733 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.048042 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.048046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.048047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.048232 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.048346 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.048702 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.048818 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:19:50.049026 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.038760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.0.0.1:56550)
0923 17:19:50.039204 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.039266 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.039412 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.039709 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.039715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.040181 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.040334 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.040343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.041019 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.041023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.042070 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.042081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.042560 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.042564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.042678 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.043291 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.043306 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.045064 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.047146 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.047158 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.047168 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.047513 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.047827 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.047832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.047835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.047936 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.048346 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.048354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.048498 (+ 144us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.048650 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.048858 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":136,"threads_started":1}
I20260923 17:19:50.050527 30524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.214.189:43711: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "65c45e932d664f29b5e591afdd6bd804"
member_type: NON_VOTER
last_known_addr {
host: "127.23.214.189"
port: 43711
}
attrs {
promote: true
}
}
dest_uuid: "e2697f8ca5a64bc591679a6e45f99c91"
cas_config_opid_index: -1
I20260923 17:19:50.051172 30524 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } }
I20260923 17:19:50.053232 31085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } }
I20260923 17:19:50.053668 31030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:50.055078 31030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Refusing update from remote peer e2697f8ca5a64bc591679a6e45f99c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 17:19:50.055248 31085 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.214.189:43711
WARNING: no policy specified for oryx/127.23.214.188@KRBTEST.COM; defaulting to no policy
I20260923 17:19:50.055847 30988 master.cc:537] Master@127.23.214.189:43711 shutting down...
I20260923 17:19:50.055914 31085 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:50.056222 31084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } }
I20260923 17:19:50.056347 31084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.188@KRBTEST.COM" created.
W20260923 17:19:50.058892 30510 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 17:19:50.060215 30510 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 -> Peer 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189:43711): Couldn't send request to peer 65c45e932d664f29b5e591afdd6bd804. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:19:50.060779 30988 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:50.061082 30988 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:50.061189 30988 tablet_replica.cc:330] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: stopping tablet replica
I20260923 17:19:50.066179 30988 master.cc:552] Master@127.23.214.189:43711 shutdown complete.
I20260923 17:19:50.066238 30988 master_runner.cc:305] Clearing existing system tablet
I20260923 17:19:50.066485 30988 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.066543 30988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.067895 30988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.068800 31091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.069043 30988 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:50.069171 30988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.069205 30988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
uuid: "65c45e932d664f29b5e591afdd6bd804"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "ead177582a915679c38934fd00c9be8f"
server_key_iv: "5d2c882b1db6241880f9eda40f2c31b5"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.071396 30988 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:50.073843 30988 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:19:50.073992 30988 log.cc:1196] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 17:19:50.074277 30988 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Deleting consensus metadata
I20260923 17:19:50.074445 30988 master_runner.cc:315] Copying system tablet from 127.23.214.190:40645
I20260923 17:19:50.075651 30988 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: tablet copy: Beginning tablet copy session from remote peer at address 127.23.214.190:40645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.188.keytab.
Entry for principal oryx/127.23.214.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.188.keytab.
I20260923 17:19:50.076753 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:32839
--webserver_interface=127.23.214.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.188.keytab
--principal=oryx/127.23.214.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:19:50.086189 31097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.076222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38244)
0923 17:19:50.076486 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.076503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.076619 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.077074 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.077077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.077287 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.077396 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.077402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.078408 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.078413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.079414 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.079429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.080244 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.080250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.080366 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.081044 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.081066 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.081588 (+ 522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.083919 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.083923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.083927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.084224 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.084574 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.084580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.084585 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.084687 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.085256 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.085261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.085390 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.085570 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.085766 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
I20260923 17:19:50.086352 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.076613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.23.214.190:40645)
0923 17:19:50.076724 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.076734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.076749 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:19:50.076905 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.076909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.076961 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.077055 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.077522 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.078261 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.079563 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.079972 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.080089 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.081772 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.081778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.081784 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.081820 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.083758 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.084351 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.084356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.084361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.084458 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.084801 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.084807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.084810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.085098 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.085205 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.086054 (+ 849us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.086192 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:19:50.086985 30544 tablet_copy_service.cc:140] P e2697f8ca5a64bc591679a6e45f99c91: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 65c45e932d664f29b5e591afdd6bd804 ({username='oryx', principal='oryx/127.23.214.189@KRBTEST.COM'} at 127.0.0.1:38244)
I20260923 17:19:50.087216 30544 tablet_copy_service.cc:161] P e2697f8ca5a64bc591679a6e45f99c91: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 65c45e932d664f29b5e591afdd6bd804 at {username='oryx', principal='oryx/127.23.214.189@KRBTEST.COM'} at 127.0.0.1:38244: session id = 65c45e932d664f29b5e591afdd6bd804-00000000000000000000000000000000
I20260923 17:19:50.089536 30544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:19:50.091640 30988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:19:50.094054 30988 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: tablet copy: Starting download of 0 data blocks...
I20260923 17:19:50.094260 30988 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: tablet copy: Starting download of 1 WAL segments...
I20260923 17:19:50.096235 30988 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:19:50.098397 30544 tablet_copy_service.cc:342] P e2697f8ca5a64bc591679a6e45f99c91: Request end of tablet copy session 65c45e932d664f29b5e591afdd6bd804-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.214.189@KRBTEST.COM'} at 127.0.0.1:38244
I20260923 17:19:50.098551 30544 tablet_copy_service.cc:433] P e2697f8ca5a64bc591679a6e45f99c91: ending tablet copy session 65c45e932d664f29b5e591afdd6bd804-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 65c45e932d664f29b5e591afdd6bd804
I20260923 17:19:50.106212 30988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:50.107901 31099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.107952 31102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.110075 31100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:50.111218 30988 server_base.cc:1034] running on GCE node
I20260923 17:19:50.111410 30988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:50.111747 30988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:50.112901 30988 hybrid_clock.cc:648] HybridClock initialized: now 1790183990112889 us; error 29 us; skew 500 ppm
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:50.114539 30988 init.cc:377] Logged in from keytab as oryx/127.23.214.189@KRBTEST.COM (short username oryx)
I20260923 17:19:50.115031 30988 webserver.cc:460] Webserver started at http://127.23.214.189:36301/ using document root <none> and password file <none>
I20260923 17:19:50.115222 30988 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.115257 30988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.116604 30988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.117638 31109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.117903 30988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.117978 30988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
uuid: "65c45e932d664f29b5e591afdd6bd804"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "ead177582a915679c38934fd00c9be8f"
server_key_iv: "5d2c882b1db6241880f9eda40f2c31b5"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.118060 30988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:50.130184 30988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:50.132515 30988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:50.137235 30988 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.189:43711
I20260923 17:19:50.137961 30988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
I20260923 17:19:50.139681 31162 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:19:50.140638 31162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Bootstrap starting.
I20260923 17:19:50.142144 31161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.189:43711 every 8 connection(s)
I20260923 17:19:50.150923 31162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:19:50.151216 31162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: Bootstrap complete.
I20260923 17:19:50.151738 31162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } }
I20260923 17:19:50.151885 31162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65c45e932d664f29b5e591afdd6bd804, State: Initialized, Role: LEARNER
I20260923 17:19:50.152024 31162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } }
I20260923 17:19:50.152439 31165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } }
I20260923 17:19:50.152546 31162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:50.152631 31165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:19:50.155526 31176 catalog_manager.cc:1259] Loaded cluster ID: 3ad1c6fb21b34c86a4afaefcd9e7447d
I20260923 17:19:50.155656 31176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: loading cluster ID for follower catalog manager: success
I20260923 17:19:50.157799 31176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: acquiring CA information for follower catalog manager: success
I20260923 17:19:50.159183 31176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:19:50.211575 31098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:50.211921 31098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:50.212013 31098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:50.216471 31098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:50.216578 31098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:50.216614 31098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:50.216655 31098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:50.216692 31098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:50.222744 31098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.188.keytab
--principal=oryx/127.23.214.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:32839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.214.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:50.223943 31098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:50.225003 31098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:50.233913 31182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.233913 31184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.233959 31181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:50.234328 31098 server_base.cc:1034] running on GCE node
I20260923 17:19:50.234915 31098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:50.236110 31098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:50.237298 31098 hybrid_clock.cc:648] HybridClock initialized: now 1790183990237286 us; error 29 us; skew 500 ppm
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:50.240103 31098 init.cc:377] Logged in from keytab as oryx/127.23.214.188@KRBTEST.COM (short username oryx)
I20260923 17:19:50.241017 31098 webserver.cc:460] Webserver started at http://127.23.214.188:44033/ using document root <none> and password file <none>
I20260923 17:19:50.241603 31098 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.241696 31098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.242009 31098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:50.244529 31098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/instance:
uuid: "f4b1fa64f73f44249b0bd6cd57dd9297"
format_stamp: "Formatted at 2026-09-23 17:19:50 on dist-test-slave-9q7f"
server_key: "aa40fc607642058bf9bd8611be51aa26"
server_key_iv: "c385e397ff6e151ceb8aaf9d502c183d"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.245271 31098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal/instance:
uuid: "f4b1fa64f73f44249b0bd6cd57dd9297"
format_stamp: "Formatted at 2026-09-23 17:19:50 on dist-test-slave-9q7f"
server_key: "aa40fc607642058bf9bd8611be51aa26"
server_key_iv: "c385e397ff6e151ceb8aaf9d502c183d"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.249370 31098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:19:50.249989 30715 heartbeater.cc:503] Master 127.23.214.190:40645 was elected leader, sending a full tablet report...
I20260923 17:19:50.252261 31191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.253783 31098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:50.253962 31098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
uuid: "f4b1fa64f73f44249b0bd6cd57dd9297"
format_stamp: "Formatted at 2026-09-23 17:19:50 on dist-test-slave-9q7f"
server_key: "aa40fc607642058bf9bd8611be51aa26"
server_key_iv: "c385e397ff6e151ceb8aaf9d502c183d"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.254096 31098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:50.271719 31098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:50.275219 31098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:50.275453 31098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:50.282873 31098 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.188:32839
I20260923 17:19:50.283171 31243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.188:32839 every 8 connection(s)
I20260923 17:19:50.284355 31098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
I20260923 17:19:50.284564 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 31098
I20260923 17:19:50.287632 31244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:50.292586 31244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.214.188@KRBTEST.COM
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.188@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:50.300767 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.286158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.23.214.188:32839)
0923 17:19:50.286583 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.286604 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.286654 (+ 50us) server_negotiation.cc:407] Connection header received
0923 17:19:50.287475 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.287492 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.287899 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.288196 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.289143 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.290365 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.291618 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.292126 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.292247 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.295284 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.295289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.295302 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.295360 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.297542 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.298054 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.298060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.298066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.298192 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.298515 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.298518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.298520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.298930 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.299156 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.299404 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.299706 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":160,"threads_started":1}
I20260923 17:19:50.305593 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.294973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.23.214.190:40645)
0923 17:19:50.295117 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.295124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.295346 (+ 222us) server_negotiation.cc:407] Connection header received
0923 17:19:50.295476 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.295479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.295537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.295630 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.296499 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.297318 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.298344 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.298575 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.298708 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.301309 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.301314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.301316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.301342 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.303759 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.304248 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.304252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.304254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.304347 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.304635 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.304639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.304641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.304828 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.304935 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.305291 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.305415 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:19:50.305629 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.294784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38258)
0923 17:19:50.295204 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.295256 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.295377 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.295691 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.295697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.296170 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.296338 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.296348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.297457 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.297460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.298201 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.298211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.298712 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.298716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.298847 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.299384 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.299396 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.301114 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.303898 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.303907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.303912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.304138 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.304453 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.304456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.304458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.304545 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.304933 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.304940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.305065 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.305224 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.305416 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":166,"threads_started":1}
I20260923 17:19:50.306716 31244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.188@KRBTEST.COM for oryx/127.23.214.189@KRBTEST.COM
I20260923 17:19:50.314990 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.307187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.0.0.1:56554)
0923 17:19:50.307375 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.307399 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.307490 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.307797 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.307799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.307940 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.308036 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.308041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.308933 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.308935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.309452 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.309459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.309921 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.309927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.310021 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.310426 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.310437 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.311735 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.313710 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.313712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.313714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.313977 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.314203 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.314205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.314207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.314276 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.314589 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.314591 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.314667 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.314751 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.314861 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:19:50.315065 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.307314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.23.214.189:43711)
0923 17:19:50.307561 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.307564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.307577 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:19:50.307613 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.307615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.307657 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.307744 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.308144 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.308840 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.309549 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.309772 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.309851 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.311897 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.311900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.311903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.311940 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.313615 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.314046 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.314049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.314050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.314124 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.314337 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.314339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.314341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.314515 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.314590 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.314792 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.314900 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
I20260923 17:19:50.315887 31244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.188@KRBTEST.COM for oryx/127.23.214.188@KRBTEST.COM
I20260923 17:19:50.324620 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.316428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.0.0.1:57666)
0923 17:19:50.316613 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.316626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.316686 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.316895 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.316897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.317018 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.317088 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.317092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.318417 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.318420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.318972 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.318980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.319497 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.319500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.319557 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.320093 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.320103 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.321368 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.323219 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.323222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.323223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.323427 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.323727 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.323729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.323730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.323810 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.324166 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.324168 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.324244 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.324326 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.324434 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":768}
I20260923 17:19:50.324627 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.316500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.23.214.188:32839)
0923 17:19:50.316669 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.316675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.316693 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:19:50.316752 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.316761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.316797 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.316865 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.317173 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.318278 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.319095 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.319363 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.319460 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.321509 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.321512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.321515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.321534 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.323104 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.323538 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.323541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.323542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.323635 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.323891 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.323893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.323894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.324082 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.324165 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.324396 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.324498 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:19:50.328629 31244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Bootstrap starting.
I20260923 17:19:50.331430 31244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:50.332235 31244 log.cc:824] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:50.334470 31244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: No bootstrap required, opened a new log
I20260923 17:19:50.337628 31244 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } }
I20260923 17:19:50.337883 31244 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:50.337992 31244 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b1fa64f73f44249b0bd6cd57dd9297, State: Initialized, Role: FOLLOWER
I20260923 17:19:50.338508 31244 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } }
I20260923 17:19:50.339037 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } } }
I20260923 17:19:50.339162 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:19:50.339841 31244 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:19:50.344925 31261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:19:50.345042 31261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:19:50.345685 31250 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:50.345856 31250 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } }
I20260923 17:19:50.346920 31250 leader_election.cc:288] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645), 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189:43711)
I20260923 17:19:50.347265 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2697f8ca5a64bc591679a6e45f99c91" is_pre_election: true
I20260923 17:19:50.347465 31130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c45e932d664f29b5e591afdd6bd804" is_pre_election: true
I20260923 17:19:50.347621 31130 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 LEARNER]: Handling vote request from an unknown peer f4b1fa64f73f44249b0bd6cd57dd9297
I20260923 17:19:50.347764 31130 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate f4b1fa64f73f44249b0bd6cd57dd9297 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260923 17:19:50.348276 31192 leader_election.cc:302] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4b1fa64f73f44249b0bd6cd57dd9297; no voters: 65c45e932d664f29b5e591afdd6bd804, e2697f8ca5a64bc591679a6e45f99c91
I20260923 17:19:50.348588 31250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:50.348997 31098 master_runner.cc:428] Detected that this master f4b1fa64f73f44249b0bd6cd57dd9297 is joining an existing cluster
I20260923 17:19:50.349038 31098 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.214.188:32839
I20260923 17:19:50.350090 31250 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:19:50.354923 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30586
I20260923 17:19:50.360527 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.351717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.23.214.188:32839)
0923 17:19:50.351816 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.351820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.352460 (+ 640us) server_negotiation.cc:407] Connection header received
0923 17:19:50.352585 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.352588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.352629 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.352675 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.353573 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.354277 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.355436 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.355759 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.355823 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.357198 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.357201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.357204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.357225 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.358914 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.359335 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.359338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.359340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.359424 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.359691 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.359694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.359696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.359868 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.359970 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.360196 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.360353 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":35}
I20260923 17:19:50.360538 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.351705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.0.0.1:57672)
0923 17:19:50.352392 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.352407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.352502 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.352806 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.352811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.353364 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.353467 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.353473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.354392 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.354395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.355320 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.355329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.355979 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.355982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.356063 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.356637 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.356648 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.357068 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.359010 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.359014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.359017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.359215 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.359518 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.359521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.359523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.359602 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.359958 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.359960 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.360040 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.360153 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.360294 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":69,"threads_started":1}
I20260923 17:19:50.366277 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:34811
--local_ip_for_outbound_sockets=127.23.214.129
--tserver_master_addrs=127.23.214.190:40645
--webserver_port=34489
--webserver_interface=127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab
--principal=oryx/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:19:50.368937 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.351263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.0.0.1:56568)
0923 17:19:50.351668 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.351685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.351782 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.352485 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.352489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.352838 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.353020 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.353030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.362587 (+ 9557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.362590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.363250 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.363258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.363817 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.363821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.363923 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.364535 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.364553 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.364920 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.367024 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.367028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.367031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.367250 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.367565 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.367569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.367572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.367667 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.368238 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.368243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.368376 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.368497 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.368677 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":6,"thread_start_us":121,"threads_started":1}
I20260923 17:19:50.369521 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.352001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.23.214.189:43711)
0923 17:19:50.352147 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.352152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.352170 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:19:50.352212 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.352215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.352276 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.352358 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.353173 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.354128 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.363404 (+ 9276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.363686 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.363819 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.365094 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.365099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.365102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.365139 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.366880 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.367365 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.367369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.367371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.367458 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.367777 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.367784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.367787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.368061 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.368171 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.369243 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.369406 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:19:50.383383 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.351304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.23.214.190:40645)
0923 17:19:50.351440 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.351446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.352180 (+ 734us) server_negotiation.cc:407] Connection header received
0923 17:19:50.352292 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.352295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.352349 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.352436 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.353168 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.354028 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.355686 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.356010 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.356078 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.379452 (+ 23374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.379457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.379460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.379496 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.381390 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.382018 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.382022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.382025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.382126 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.382427 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.382430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.382432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.382672 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.382773 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.383085 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.383214 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:19:50.383401 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.351274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38260)
0923 17:19:50.352082 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.352094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.352213 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.352557 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.352563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.352841 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.353035 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.353042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.354150 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.354153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.355576 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.355583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.356229 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.356233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.356323 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.378567 (+ 22244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.378587 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.379093 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.381561 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.381566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.381568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.381913 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.382228 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.382231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.382233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.382330 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.382783 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.382786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.382907 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.383014 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.383186 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":36,"spinlock_wait_cycles":44928,"thread_start_us":144,"threads_started":1}
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183988, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.190@KRBTEST.COM for oryx/127.23.214.188@KRBTEST.COM
I20260923 17:19:50.398274 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.388525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.23.214.188:32839)
0923 17:19:50.388640 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.388645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.388657 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:19:50.388741 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.388744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.388795 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.388866 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.389431 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.390306 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.391570 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.391929 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.392033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.394654 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.394658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.394660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.394688 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.396307 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.396830 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.396834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.396836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.396926 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.397215 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.397218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.397219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.397439 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.397525 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.397993 (+ 468us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.398106 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:19:50.399006 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.388001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.0.0.1:57688)
0923 17:19:50.388247 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.388263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.388352 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.388994 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.388997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.389198 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.389307 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.389314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.390438 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.390442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.391446 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.391458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.392124 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.392128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.392271 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.392802 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.392817 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.394483 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.396441 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.396444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.396447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.396706 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.397029 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.397032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.397034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.397120 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.397569 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.397572 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.397669 (+ 97us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.397778 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.398856 (+ 1078us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260923 17:19:50.400545 30524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.214.188:32839: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297"
member_type: NON_VOTER
last_known_addr {
host: "127.23.214.188"
port: 32839
}
attrs {
promote: true
}
}
dest_uuid: "e2697f8ca5a64bc591679a6e45f99c91"
cas_config_opid_index: 5
I20260923 17:19:50.401161 30524 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } }
I20260923 17:19:50.402510 31085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.403970 31085 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.214.188:32839
I20260923 17:19:50.404146 31085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.404235 31085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:50.404932 31218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Refusing update from remote peer e2697f8ca5a64bc591679a6e45f99c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:19:50.405606 31084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:50.409113 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.404487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.23.214.189:43711)
0923 17:19:50.404610 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.404615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.404629 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:19:50.404681 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.404684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.404760 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.404852 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:50.405229 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.406044 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.407542 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.408190 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.408288 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.408441 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.408853 (+ 412us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.408940 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:19:50.409559 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.402765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.0.0.1:56572)
0923 17:19:50.403076 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.403092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.403166 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.404986 (+ 1820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.404989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.405009 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:50.405101 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.405107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.406190 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.406194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.407392 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.407403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.408546 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.408549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.408662 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.408710 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.409331 (+ 621us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.409437 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260923 17:19:50.409713 31218 raft_consensus.cc:858] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } }
W20260923 17:19:50.409806 31218 raft_consensus.cc:1565] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } }
I20260923 17:19:50.410135 31130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 LEARNER]: Refusing update from remote peer e2697f8ca5a64bc591679a6e45f99c91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:19:50.410552 31084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 17:19:50.411314 31218 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } }
I20260923 17:19:50.412379 31130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.413127 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2697f8ca5a64bc591679a6e45f99c91. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } }
I20260923 17:19:50.413223 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 17:19:50.413293 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } }
I20260923 17:19:50.413347 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 17:19:50.413401 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } }
I20260923 17:19:50.413451 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 17:19:50.414152 31098 master.cc:537] Master@127.23.214.188:32839 shutting down...
I20260923 17:19:50.415709 31165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2697f8ca5a64bc591679a6e45f99c91. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.415849 31165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:19:50.416010 31276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.416226 31276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:19:50.419765 31271 mvcc.cc:205] Tried to move back new op lower bound from 7332593618830569472 to 7332593618647506944. Current Snapshot: MvccSnapshot[applied={T|T < 7332593618684710912}]
I20260923 17:19:50.419987 31084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: NON_VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: true } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.420128 31084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:50.420821 31085 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: attempting to promote NON_VOTER 65c45e932d664f29b5e591afdd6bd804 to VOTER
I20260923 17:19:50.421209 31085 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } }
I20260923 17:19:50.422887 31130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 LEARNER]: Refusing update from remote peer e2697f8ca5a64bc591679a6e45f99c91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 17:19:50.423267 31086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 17:19:50.425079 31098 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260923 17:19:50.425228 30510 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 -> Peer f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188:32839): Couldn't send request to peer f4b1fa64f73f44249b0bd6cd57dd9297. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:19:50.425460 31098 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260923 17:19:50.425577 31098 tablet_replica.cc:330] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: stopping tablet replica
I20260923 17:19:50.426100 31086 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.427596 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.427692 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:50.427796 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.427855 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:50.428776 31130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.430452 31276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.430557 31276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:19:50.443328 31098 master.cc:552] Master@127.23.214.188:32839 shutdown complete.
I20260923 17:19:50.443405 31098 master_runner.cc:305] Clearing existing system tablet
I20260923 17:19:50.443655 31098 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.443712 31098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.445122 31098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.446166 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.446394 31098 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:50.446516 31098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.446556 31098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
uuid: "f4b1fa64f73f44249b0bd6cd57dd9297"
format_stamp: "Formatted at 2026-09-23 17:19:50 on dist-test-slave-9q7f"
server_key: "aa40fc607642058bf9bd8611be51aa26"
server_key_iv: "c385e397ff6e151ceb8aaf9d502c183d"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.448525 31098 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:50.451418 31098 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:19:50.451521 31098 log.cc:1196] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 17:19:50.451849 31098 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Deleting consensus metadata
I20260923 17:19:50.452072 31098 master_runner.cc:315] Copying system tablet from 127.23.214.190:40645
I20260923 17:19:50.453239 31098 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: tablet copy: Beginning tablet copy session from remote peer at address 127.23.214.190:40645
I20260923 17:19:50.462821 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.454286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.23.214.190:40645)
0923 17:19:50.454415 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.454420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.454436 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:19:50.454485 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.454488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.454534 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.454648 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.455252 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.456092 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.457269 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.457616 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.457723 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.459116 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.459120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.459124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.459150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.460962 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.461465 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.461469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.461471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.461562 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.461866 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.461869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.461891 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.462137 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.462244 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.462530 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.462658 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:19:50.462816 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.453932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38272)
0923 17:19:50.454202 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.454218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.454329 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.454782 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.454786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.455006 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.455114 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.455121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.456222 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.456226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.457127 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.457138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.457738 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.457742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.457838 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.458478 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.458493 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.458957 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.461087 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.461091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.461094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.461346 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.461661 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.461664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.461667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.461762 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.462241 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.462244 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.462348 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.462456 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.462607 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260923 17:19:50.463919 30544 tablet_copy_service.cc:140] P e2697f8ca5a64bc591679a6e45f99c91: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f4b1fa64f73f44249b0bd6cd57dd9297 ({username='oryx', principal='oryx/127.23.214.188@KRBTEST.COM'} at 127.0.0.1:38272)
I20260923 17:19:50.464032 30544 tablet_copy_service.cc:161] P e2697f8ca5a64bc591679a6e45f99c91: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f4b1fa64f73f44249b0bd6cd57dd9297 at {username='oryx', principal='oryx/127.23.214.188@KRBTEST.COM'} at 127.0.0.1:38272: session id = f4b1fa64f73f44249b0bd6cd57dd9297-00000000000000000000000000000000
I20260923 17:19:50.465381 30544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:19:50.466890 31098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:19:50.469275 31098 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: tablet copy: Starting download of 0 data blocks...
I20260923 17:19:50.469455 31098 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: tablet copy: Starting download of 1 WAL segments...
I20260923 17:19:50.473819 31098 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:19:50.476123 30544 tablet_copy_service.cc:342] P e2697f8ca5a64bc591679a6e45f99c91: Request end of tablet copy session f4b1fa64f73f44249b0bd6cd57dd9297-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.214.188@KRBTEST.COM'} at 127.0.0.1:38272
I20260923 17:19:50.476233 30544 tablet_copy_service.cc:433] P e2697f8ca5a64bc591679a6e45f99c91: ending tablet copy session f4b1fa64f73f44249b0bd6cd57dd9297-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f4b1fa64f73f44249b0bd6cd57dd9297
I20260923 17:19:50.485852 31098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:50.487854 31291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.488186 31288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:50.488446 31098 server_base.cc:1034] running on GCE node
W20260923 17:19:50.490087 31289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:50.490330 31098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:50.490674 31098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:50.491840 31098 hybrid_clock.cc:648] HybridClock initialized: now 1790183990491816 us; error 45 us; skew 500 ppm
I20260923 17:19:50.492650 30849 heartbeater.cc:503] Master 127.23.214.190:40645 was elected leader, sending a full tablet report...
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:50.494549 31098 init.cc:377] Logged in from keytab as oryx/127.23.214.188@KRBTEST.COM (short username oryx)
I20260923 17:19:50.495085 31098 webserver.cc:460] Webserver started at http://127.23.214.188:36487/ using document root <none> and password file <none>
I20260923 17:19:50.495306 31098 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.495347 31098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.496853 31098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.498034 31298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.498348 31098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:50.498468 31098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
uuid: "f4b1fa64f73f44249b0bd6cd57dd9297"
format_stamp: "Formatted at 2026-09-23 17:19:50 on dist-test-slave-9q7f"
server_key: "aa40fc607642058bf9bd8611be51aa26"
server_key_iv: "c385e397ff6e151ceb8aaf9d502c183d"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.498621 31098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:50.517504 31098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:50.519768 31098 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260923 17:19:50.519675 31270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:50.519992 31270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:50.520087 31270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:19:50.524916 31098 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.188:32839
I20260923 17:19:50.525365 31353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.188:32839 every 8 connection(s)
I20260923 17:19:50.525861 31098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
W20260923 17:19:50.526011 31270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:50.526135 31270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:50.526383 31270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:50.527845 31354 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:19:50.529126 31354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Bootstrap starting.
I20260923 17:19:50.532650 31270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab
--principal=oryx/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:34811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=34489
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:50.533828 31270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:50.534973 31270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:19:50.537295 31354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:19:50.537621 31354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Bootstrap complete.
I20260923 17:19:50.538194 31354 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } }
I20260923 17:19:50.538331 31354 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f4b1fa64f73f44249b0bd6cd57dd9297, State: Initialized, Role: LEARNER
I20260923 17:19:50.538429 31354 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } }
I20260923 17:19:50.538806 31357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.538895 31354 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:50.541980 31368 catalog_manager.cc:1259] Loaded cluster ID: 3ad1c6fb21b34c86a4afaefcd9e7447d
I20260923 17:19:50.542073 31368 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: loading cluster ID for follower catalog manager: success
I20260923 17:19:50.538987 31357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:19:50.543763 31368 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: acquiring CA information for follower catalog manager: success
I20260923 17:19:50.544917 31368 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:19:50.547833 31371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.547840 31373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.547888 31370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:50.548308 31270 server_base.cc:1034] running on GCE node
I20260923 17:19:50.548739 31270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:50.549495 31270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:50.550706 31270 hybrid_clock.cc:648] HybridClock initialized: now 1790183990550685 us; error 64 us; skew 500 ppm
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:50.553747 31270 init.cc:377] Logged in from keytab as oryx/127.23.214.129@KRBTEST.COM (short username oryx)
I20260923 17:19:50.554750 31270 webserver.cc:460] Webserver started at http://127.23.214.129:34489/ using document root <none> and password file <none>
I20260923 17:19:50.555279 31270 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.555348 31270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.559386 31270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:19:50.561408 31380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.562747 31270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:50.562875 31270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "cb655b09bcae49eaaf4c91acf1506549"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "4d29ef533aa89681cb5d59d3e0546af3"
server_key_iv: "96b1b0be0ce7449d5518caba67420b11"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.563444 31270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:50.579021 31270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:50.582085 31270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:50.582293 31270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:50.582923 31270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:50.583849 31270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:50.583900 31270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.583941 31270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:50.583999 31270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.593500 31270 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:34811
I20260923 17:19:50.593518 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:34811 every 8 connection(s)
I20260923 17:19:50.594638 31270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:50.597460 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 31270
I20260923 17:19:50.597654 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30720
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.189@KRBTEST.COM
I20260923 17:19:50.607522 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:35989
--local_ip_for_outbound_sockets=127.23.214.130
--tserver_master_addrs=127.23.214.190:40645
--webserver_port=37947
--webserver_interface=127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab
--principal=oryx/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.188@KRBTEST.COM
I20260923 17:19:50.612460 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.596455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:52611 (local address 127.23.214.189:43711)
0923 17:19:50.596627 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.596632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.597538 (+ 906us) server_negotiation.cc:407] Connection header received
0923 17:19:50.599069 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.599072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.599125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.599235 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.600970 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.601481 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.603428 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.603882 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.603979 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.607659 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.607665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.607671 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.607707 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.609480 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.610205 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.610211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.610213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.610306 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.610748 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.610754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.610759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.611008 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.611112 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.612115 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.612254 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:19:50.612635 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.597031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:60025 (local address 127.23.214.190:40645)
0923 17:19:50.597179 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.597184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.602575 (+ 5391us) server_negotiation.cc:407] Connection header received
0923 17:19:50.602714 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.602720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.602780 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.602860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.603517 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.604325 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.605508 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.605843 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.605968 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.608958 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.608965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.608967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.608994 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.610661 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.611160 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.611166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.611171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.611258 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.611570 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.611576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.611580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.611821 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.611909 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.612295 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.612437 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:19:50.613778 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.599025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:52563 (local address 127.23.214.188:32839)
0923 17:19:50.599288 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.599293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.602925 (+ 3632us) server_negotiation.cc:407] Connection header received
0923 17:19:50.603095 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.603099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.603157 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.603222 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.603787 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.604828 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.605974 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.606309 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.606400 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.609153 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.609158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.609162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.609193 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.612035 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.612493 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.612500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.612502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.612585 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.612889 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.612892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.612894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.613104 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.613170 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.613507 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.613629 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260923 17:19:50.614236 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.596810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.23.214.129:52611)
0923 17:19:50.597365 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.597425 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.598820 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.599489 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.599501 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.600051 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.600805 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.600827 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.601610 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.601617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.603277 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.603291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.604195 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.604201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.604460 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.605233 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.605260 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.607454 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.609619 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.609628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.609649 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.610077 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.610420 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.610425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.610430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.610613 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.611447 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.611457 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.611808 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.612631 (+ 823us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.613012 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"thread_start_us":146,"threads_started":1}
I20260923 17:19:50.614249 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.596817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:60025)
0923 17:19:50.602466 (+ 5649us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.602480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.602603 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.602984 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.602988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.603223 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.603377 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.603384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.604468 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.604471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.605380 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.605389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.606112 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.606115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.606203 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.606704 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.606718 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.608792 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.610775 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.610779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.610783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.611046 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.611370 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.611375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.611380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.611470 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.611985 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.611991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.612097 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.613007 (+ 910us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.613187 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5576,"mutex_wait_us":83,"spinlock_wait_cycles":43776,"thread_start_us":147,"threads_started":1}
I20260923 17:19:50.614529 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.598060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.23.214.129:52563)
0923 17:19:50.602835 (+ 4775us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.602849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.602984 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.603368 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.603371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.603548 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.603659 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.603665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.604971 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.604974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.605813 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.605822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.606588 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.606593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.606722 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.607142 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.607162 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.609006 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.612147 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.612150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.612152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.612380 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.612697 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.612699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.612701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.612779 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.613238 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.613240 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.613323 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.614212 (+ 889us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.614373 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4693,"thread_start_us":83,"threads_started":1}
I20260923 17:19:50.615993 31495 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40645
I20260923 17:19:50.616272 31495 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:50.616479 31494 heartbeater.cc:347] Connected to a master server at 127.23.214.188:32839
I20260923 17:19:50.616562 31494 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:50.616899 31494 heartbeater.cc:511] Master 127.23.214.188:32839 requested a full tablet report, sending...
I20260923 17:19:50.616902 31495 heartbeater.cc:511] Master 127.23.214.190:40645 requested a full tablet report, sending...
I20260923 17:19:50.616930 31496 heartbeater.cc:347] Connected to a master server at 127.23.214.189:43711
I20260923 17:19:50.617569 31496 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:50.617755 31496 heartbeater.cc:511] Master 127.23.214.189:43711 requested a full tablet report, sending...
I20260923 17:19:50.618322 30524 ts_manager.cc:194] Re-registered known tserver with Master: cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811)
I20260923 17:19:50.619009 30524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:60025
I20260923 17:19:50.619103 31118 ts_manager.cc:194] Registered new tserver with Master: cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811)
I20260923 17:19:50.619903 31318 ts_manager.cc:194] Registered new tserver with Master: cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811)
W20260923 17:19:50.732973 31502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:50.733261 31502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:50.733343 31502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:50.737479 31502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:50.737560 31502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:50.737666 31502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:50.741201 30983 heartbeater.cc:503] Master 127.23.214.190:40645 was elected leader, sending a full tablet report...
I20260923 17:19:50.743603 31502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab
--principal=oryx/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:35989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=37947
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:50.744709 31502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:50.745764 31502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:50.755774 31507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.755826 31510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.755774 31508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:50.756111 31502 server_base.cc:1034] running on GCE node
I20260923 17:19:50.756521 31502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:50.757256 31502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:50.758461 31502 hybrid_clock.cc:648] HybridClock initialized: now 1790183990758439 us; error 31 us; skew 500 ppm
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:50.761433 31502 init.cc:377] Logged in from keytab as oryx/127.23.214.130@KRBTEST.COM (short username oryx)
I20260923 17:19:50.762398 31502 webserver.cc:460] Webserver started at http://127.23.214.130:37947/ using document root <none> and password file <none>
I20260923 17:19:50.762953 31502 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.763033 31502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.767197 31502 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 17:19:50.769454 31517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.770818 31502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:19:50.770958 31502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "b5b3cb4911834d8287091fdbdac42d2a"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "a301525eb7d2d30ff2a18fbcc4e66e0c"
server_key_iv: "6554709c7911af74f03ab35bbe951259"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.771521 31502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:50.787317 31502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:50.790800 31502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:50.791029 31502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:50.791652 31502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:50.792512 31502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:50.792558 31502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.792625 31502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:50.792657 31502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.801968 31502 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:35989
I20260923 17:19:50.801985 31630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:35989 every 8 connection(s)
I20260923 17:19:50.803025 31502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:50.805857 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 31502
I20260923 17:19:50.806054 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30854
I20260923 17:19:50.813251 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:41939
--local_ip_for_outbound_sockets=127.23.214.131
--tserver_master_addrs=127.23.214.190:40645
--webserver_port=42059
--webserver_interface=127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab
--principal=oryx/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.189@KRBTEST.COM
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.188@KRBTEST.COM
I20260923 17:19:50.820765 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.804923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:54557 (local address 127.23.214.190:40645)
0923 17:19:50.805102 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.805107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.806209 (+ 1102us) server_negotiation.cc:407] Connection header received
0923 17:19:50.807671 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.807678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.807754 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.807848 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.809235 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.810221 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.811975 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.812319 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.812402 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.815848 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.815854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.815859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.815891 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.817718 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.818484 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.818490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.818495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.818592 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.819093 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.819099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.819103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.819375 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.819472 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.820348 (+ 876us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.820503 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:19:50.821241 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.806923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:54921 (local address 127.23.214.189:43711)
0923 17:19:50.807057 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.807064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.810975 (+ 3911us) server_negotiation.cc:407] Connection header received
0923 17:19:50.811154 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.811157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.811207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.811258 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.811823 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.812570 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.813771 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.814132 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.814229 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.816983 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.816987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.816990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.817019 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.819076 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.819681 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.819684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.819687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.819766 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.820094 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.820097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.820099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.820326 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.820409 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.820924 (+ 515us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.821045 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:19:50.822793 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.805333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:54557)
0923 17:19:50.806021 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.806092 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.807403 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.808058 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.808070 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.808582 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.809080 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.809092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.810360 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.810374 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.811853 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.811861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.812605 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.812611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.812852 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.813556 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.813581 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.815630 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.817864 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.817894 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.817915 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.818357 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.818720 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.818738 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.818743 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.818946 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.819600 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.819609 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.820002 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.821016 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.821545 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":1,"thread_start_us":751,"threads_started":1}
I20260923 17:19:50.823014 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.805579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:34361 (local address 127.23.214.188:32839)
0923 17:19:50.806462 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.806467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.810686 (+ 4219us) server_negotiation.cc:407] Connection header received
0923 17:19:50.810788 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.810794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.810860 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.810943 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:50.811661 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.812479 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.813853 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.814291 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.814383 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.817624 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:50.817630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:50.817635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:50.817671 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:50.819501 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.820020 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.820026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.820031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.820112 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:50.820408 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:50.820411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:50.820414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:50.820618 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:50.820690 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.822677 (+ 1987us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.822820 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795}
I20260923 17:19:50.823069 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.806531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.23.214.130:54921)
0923 17:19:50.810887 (+ 4356us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.810898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.811036 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.811384 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.811388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.811572 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.811682 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.811688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.812696 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.812698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.813631 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.813644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.814402 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.814407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.814509 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.815258 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.815273 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.816855 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.819206 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.819210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.819213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.819576 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.819892 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.819894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.819896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.819992 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.820479 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.820482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.820737 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.821386 (+ 649us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.821606 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4283,"mutex_wait_us":129,"spinlock_wait_cycles":896,"thread_start_us":101,"threads_started":1}
I20260923 17:19:50.824462 31633 heartbeater.cc:347] Connected to a master server at 127.23.214.189:43711
I20260923 17:19:50.824523 31632 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40645
I20260923 17:19:50.824707 31632 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:50.824707 31633 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:50.825317 31632 heartbeater.cc:511] Master 127.23.214.190:40645 requested a full tablet report, sending...
I20260923 17:19:50.825318 31633 heartbeater.cc:511] Master 127.23.214.189:43711 requested a full tablet report, sending...
I20260923 17:19:50.826658 30524 ts_manager.cc:194] Re-registered known tserver with Master: b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
I20260923 17:19:50.826790 31118 ts_manager.cc:194] Registered new tserver with Master: b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
I20260923 17:19:50.827150 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.821308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.0.0.1:57704)
0923 17:19:50.821690 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.821705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.821779 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.823425 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.823428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.823446 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:50.823552 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.823564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.824598 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.824601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.825688 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.825698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.826787 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.826790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.826870 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.826891 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.826962 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.827031 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260923 17:19:50.827356 30524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:54557
I20260923 17:19:50.827482 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.821969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.23.214.188:32839)
0923 17:19:50.823184 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:50.823188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:50.823201 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:19:50.823251 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:50.823254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:50.823297 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:50.823410 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:50.823679 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.824473 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.825823 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.826484 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.826596 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.826731 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:50.827287 (+ 556us) server_negotiation.cc:299] Negotiation successful
0923 17:19:50.827355 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":106,"threads_started":1}
I20260923 17:19:50.828050 31328 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 17:19:50.829489 31357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2697f8ca5a64bc591679a6e45f99c91. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:50.829689 31357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:19:50.830402 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:50.805333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.23.214.130:34361)
0923 17:19:50.810567 (+ 5234us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:50.810584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:50.810686 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:50.811090 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:50.811096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:50.811345 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:50.811508 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.811519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.812646 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.812652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.813689 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:50.813703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:50.814583 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:50.814589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:50.814739 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:50.815383 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:50.815401 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:50.817447 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:50.819649 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.819655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.819660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.819914 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.820225 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:50.820228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:50.820229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:50.820302 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:50.822409 (+ 2107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:50.822411 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:50.822498 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 17:19:50.828122 (+ 5624us) client_negotiation.cc:240] Negotiation successful
0923 17:19:50.830227 (+ 2105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5144,"mutex_wait_us":108,"spinlock_wait_cycles":33536,"thread_start_us":186,"threads_started":1}
I20260923 17:19:50.830955 31631 heartbeater.cc:347] Connected to a master server at 127.23.214.188:32839
I20260923 17:19:50.831053 31631 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:50.831267 31631 heartbeater.cc:511] Master 127.23.214.188:32839 requested a full tablet report, sending...
I20260923 17:19:50.832022 31318 ts_manager.cc:194] Registered new tserver with Master: b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
W20260923 17:19:50.936421 31638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:50.936741 31638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:50.936787 31638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:50.941340 31638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:50.941418 31638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:50.941515 31638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:50.947263 31638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab
--principal=oryx/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:41939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=42059
--tserver_master_addrs=127.23.214.190:40645,127.23.214.189:43711,127.23.214.188:32839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:50.948455 31638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:50.949553 31638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:50.960090 31647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.960073 31644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:50.960155 31645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:50.960243 31638 server_base.cc:1034] running on GCE node
I20260923 17:19:50.960812 31638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:50.961584 31638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:50.962743 31638 hybrid_clock.cc:648] HybridClock initialized: now 1790183990962716 us; error 39 us; skew 500 ppm
Sep 23 17:19:50 dist-test-slave-9q7f krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:50.965629 31638 init.cc:377] Logged in from keytab as oryx/127.23.214.131@KRBTEST.COM (short username oryx)
I20260923 17:19:50.966600 31638 webserver.cc:460] Webserver started at http://127.23.214.131:42059/ using document root <none> and password file <none>
I20260923 17:19:50.967167 31638 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:50.967243 31638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:50.971414 31638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:19:50.973733 31654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:50.975201 31638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:50.975379 31638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "c37d82581dd14b6db7223f55d05e2da0"
format_stamp: "Formatted at 2026-09-23 17:19:49 on dist-test-slave-9q7f"
server_key: "10779e60a7e9d74556a2f2d4a062096e"
server_key_iv: "d049f9c0e7e731958200eae058132849"
server_key_version: "encryptionkey@0"
I20260923 17:19:50.976017 31638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:51.001397 31638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:51.004864 31638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:51.005080 31638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:51.005704 31638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:19:51.006661 31638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:51.006733 31638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:51.006808 31638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:51.006852 31638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:51.016075 31638 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:41939
I20260923 17:19:51.016099 31767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:41939 every 8 connection(s)
I20260923 17:19:51.017084 31638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:51.022274 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 31638
Sep 23 17:19:51 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.189@KRBTEST.COM
Sep 23 17:19:51 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.188@KRBTEST.COM
Sep 23 17:19:51 dist-test-slave-9q7f krb5kdc[30479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183990, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:51.037667 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.020569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:46891 (local address 127.23.214.188:32839)
0923 17:19:51.020690 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.020694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.020711 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:19:51.021276 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.021279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.021334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.021394 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.023448 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.024273 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.026475 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.026859 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.026949 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.033329 (+ 6380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.033333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.033336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.033367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.035248 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.035901 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.035905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.035907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.035993 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.036392 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.036395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.036397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.036609 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.036685 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.037364 (+ 679us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.037492 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:19:51.039345 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.024903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.23.214.189:43711)
0923 17:19:51.025922 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.025927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.025940 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:19:51.025975 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.025978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.026019 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.026109 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.028085 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.029139 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.031292 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.031831 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.031993 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.035019 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.035023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.035026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.035055 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.036970 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.037498 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.037501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.037503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.037592 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.037913 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.037920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.037924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.038186 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.038342 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.038953 (+ 611us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.039105 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":94,"threads_started":1}
I20260923 17:19:51.040112 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.020378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.23.214.131:46891)
0923 17:19:51.020644 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.020658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.021046 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.021736 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.021742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.022476 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:51.023270 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.023294 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.024417 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.024420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.026329 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.026341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.027124 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.027127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.027226 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.027631 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:51.027650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:51.033164 (+ 5514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:51.035389 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:51.035396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:51.035417 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:51.035770 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:51.036092 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:51.036095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:51.036097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:51.036289 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:51.036748 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:51.036755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:51.037095 (+ 340us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.038191 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.038584 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":163,"spinlock_wait_cycles":57600,"thread_start_us":113,"threads_started":1}
I20260923 17:19:51.041566 31768 heartbeater.cc:347] Connected to a master server at 127.23.214.188:32839
I20260923 17:19:51.041813 31768 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:51.042435 31768 heartbeater.cc:511] Master 127.23.214.188:32839 requested a full tablet report, sending...
I20260923 17:19:51.045674 31318 ts_manager.cc:194] Registered new tserver with Master: c37d82581dd14b6db7223f55d05e2da0 (127.23.214.131:41939)
I20260923 17:19:51.046598 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.018922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:46993 (local address 127.23.214.190:40645)
0923 17:19:51.019039 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.019045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.020270 (+ 1225us) server_negotiation.cc:407] Connection header received
0923 17:19:51.021288 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.021291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.021347 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.021400 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.023459 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.024189 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.025836 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.026194 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.026288 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.031825 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.031829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.031832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.031862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.044318 (+ 12456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.044853 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.044857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.044859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.044945 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.045210 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.045212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.045214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.045419 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.045508 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.046320 (+ 812us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.046449 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 17:19:51.047022 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.019355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:46993)
0923 17:19:51.020180 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.020194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.021046 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.021734 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.021740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.022453 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:51.023266 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.023291 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.024341 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.024344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.025694 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.025704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.026556 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.026560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.026719 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.027866 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:51.027882 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:51.031427 (+ 3545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:51.044445 (+ 13018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:51.044448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:51.044450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:51.044713 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:51.045037 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:51.045039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:51.045041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:51.045121 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:51.045551 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:51.045554 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:51.045644 (+ 90us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.045745 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.046852 (+ 1107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":307,"spinlock_wait_cycles":15872,"thread_start_us":144,"threads_started":1}
I20260923 17:19:51.047036 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.018888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:54461 (local address 127.23.214.189:43711)
0923 17:19:51.019028 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.019032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.020040 (+ 1008us) server_negotiation.cc:407] Connection header received
0923 17:19:51.021275 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.021279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.021334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.021392 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.023451 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.024269 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.025832 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.026194 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.026288 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.032602 (+ 6314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.032606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.032608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.032633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.044671 (+ 12038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.045217 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.045220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.045223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.045310 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.045660 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.045666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.045669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.045942 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.046340 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.046743 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.046879 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:19:51.047621 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.019355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.23.214.131:54461)
0923 17:19:51.019865 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.019916 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.021046 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.021736 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.021755 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.022476 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:51.023267 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.023289 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.024417 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.024420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.025698 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.025704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.026468 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.026472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.026715 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.027495 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:51.027520 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:51.032450 (+ 4930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:51.044825 (+ 12375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:51.044829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:51.044831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:51.045106 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:51.045442 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:51.045445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:51.045447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:51.045543 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:51.046392 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:51.046396 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:51.046493 (+ 97us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.046596 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.047418 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":837,"spinlock_wait_cycles":7168,"thread_start_us":558,"threads_started":1}
I20260923 17:19:51.047633 31769 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40645
I20260923 17:19:51.047752 31769 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:51.047940 31769 heartbeater.cc:511] Master 127.23.214.190:40645 requested a full tablet report, sending...
I20260923 17:19:51.048589 30524 ts_manager.cc:194] Re-registered known tserver with Master: c37d82581dd14b6db7223f55d05e2da0 (127.23.214.131:41939)
I20260923 17:19:51.049141 30524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:46993
I20260923 17:19:51.049796 31770 heartbeater.cc:347] Connected to a master server at 127.23.214.189:43711
I20260923 17:19:51.049873 31770 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:51.050096 31770 heartbeater.cc:511] Master 127.23.214.189:43711 requested a full tablet report, sending...
I20260923 17:19:51.052614 31118 ts_manager.cc:194] Registered new tserver with Master: c37d82581dd14b6db7223f55d05e2da0 (127.23.214.131:41939)
I20260923 17:19:51.055645 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.028447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.23.214.190:40645)
0923 17:19:51.028733 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.028737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.041176 (+ 12439us) server_negotiation.cc:407] Connection header received
0923 17:19:51.041289 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.041292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.041351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.041485 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.042359 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.043229 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.048944 (+ 5715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.049416 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.049528 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.051527 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.051531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.051534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.051560 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.053457 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.054033 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.054037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.054039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.054130 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.054601 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.054604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.054606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.054813 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.054962 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.055300 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.055431 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260923 17:19:51.056363 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.025277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.23.214.188:32839)
0923 17:19:51.025382 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.025386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.029507 (+ 4121us) server_negotiation.cc:407] Connection header received
0923 17:19:51.029700 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.029704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.029763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.029824 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.031064 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.042789 (+ 11725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.045025 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.045431 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.045528 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.049742 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.049746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.049749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.049778 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.054259 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.054896 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.054901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.054903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.054991 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.055340 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.055344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.055346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.055601 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.055709 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.056103 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.056223 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:19:51.059010 30524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:51.062448 30524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:51.064886 31084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: NON_VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: true } } }
I20260923 17:19:51.065038 31084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:51.066808 31084 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: attempting to promote NON_VOTER f4b1fa64f73f44249b0bd6cd57dd9297 to VOTER
I20260923 17:19:51.067193 31084 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:51.068434 31130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Refusing update from remote peer e2697f8ca5a64bc591679a6e45f99c91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 17:19:51.068434 31328 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 LEARNER]: Refusing update from remote peer e2697f8ca5a64bc591679a6e45f99c91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 17:19:51.068797 31786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:19:51.068936 31785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:19:51.070760 31785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.071200 31130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.071169 31328 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.072269 31785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.072358 31785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:51.072481 31785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.072553 31785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:51.072643 31785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.072728 31785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:51.072700 31357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.072829 31357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:19:51.073396 31794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2697f8ca5a64bc591679a6e45f99c91" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:51.073516 31794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:19:51.085279 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.078986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:41939 (local address 127.0.0.1:53724)
0923 17:19:51.079275 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.079291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.079398 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.080261 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.080264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.080284 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:51.080430 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.080437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.082062 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.082065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.083667 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.083678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.084808 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.084814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.084903 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.084954 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.085013 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.085090 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":239,"spinlock_wait_cycles":8320,"thread_start_us":111,"threads_started":1}
I20260923 17:19:51.085296 31796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.078666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:34811 (local address 127.0.0.1:45416)
0923 17:19:51.079231 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.079246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.079394 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.080263 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.080266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.080302 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:51.080710 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.080717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.082059 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.082062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.083667 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.083678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.084895 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.084901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.085012 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.085031 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.085098 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.085160 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":455,"spinlock_wait_cycles":69504}
I20260923 17:19:51.085429 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.080243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.23.214.130:35989)
0923 17:19:51.080843 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.080855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.080878 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:19:51.080949 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.080954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.081147 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.081297 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:51.081495 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.082511 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.083553 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.084396 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.084514 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.084665 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.085168 (+ 503us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.085264 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":114,"threads_started":1}
I20260923 17:19:51.085724 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.078944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.23.214.129:34811)
0923 17:19:51.079716 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.079729 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.079747 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:19:51.079848 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.079853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.080082 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.080420 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:51.080859 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.081932 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.083822 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.084769 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.084893 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.085027 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.085127 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.085241 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":88,"threads_started":1}
I20260923 17:19:51.086311 31797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.079301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.23.214.131:41939)
0923 17:19:51.079631 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.079650 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.079677 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:19:51.079786 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.079794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.080100 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.080582 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:51.080725 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.081937 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.083827 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.084647 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.085904 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.086038 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.086087 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.086174 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
I20260923 17:19:51.086516 31798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.079849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:35989 (local address 127.0.0.1:51246)
0923 17:19:51.080161 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.080177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.080259 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.081273 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.081283 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.081302 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:51.081402 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.081408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.082630 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.082633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.083424 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.083437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.084719 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.084722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.084795 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.084814 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.084910 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.084995 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":23,"thread_start_us":104,"threads_started":1}
I20260923 17:19:51.088919 31702 tablet_service.cc:1467] Processing CreateTablet for tablet 49567b0ed6c24355ac2f8c70b40c4de8 (DEFAULT_TABLE table=test-table [id=f2132eea8f5b4fcf89d8e82f602d221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:51.088920 31428 tablet_service.cc:1467] Processing CreateTablet for tablet 49567b0ed6c24355ac2f8c70b40c4de8 (DEFAULT_TABLE table=test-table [id=f2132eea8f5b4fcf89d8e82f602d221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:51.089917 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49567b0ed6c24355ac2f8c70b40c4de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:51.089932 31428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49567b0ed6c24355ac2f8c70b40c4de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:51.091925 31565 tablet_service.cc:1467] Processing CreateTablet for tablet 49567b0ed6c24355ac2f8c70b40c4de8 (DEFAULT_TABLE table=test-table [id=f2132eea8f5b4fcf89d8e82f602d221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:51.092958 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49567b0ed6c24355ac2f8c70b40c4de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:51.096808 31800 tablet_bootstrap.cc:492] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Bootstrap starting.
I20260923 17:19:51.097857 31801 tablet_bootstrap.cc:492] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Bootstrap starting.
I20260923 17:19:51.099287 31800 tablet_bootstrap.cc:654] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:51.100171 31801 tablet_bootstrap.cc:654] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:51.100298 31800 log.cc:824] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:51.101013 31801 log.cc:824] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:51.102010 31802 tablet_bootstrap.cc:492] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Bootstrap starting.
I20260923 17:19:51.102651 31800 tablet_bootstrap.cc:492] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: No bootstrap required, opened a new log
I20260923 17:19:51.102869 31800 ts_tablet_manager.cc:1397] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:19:51.102962 31801 tablet_bootstrap.cc:492] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: No bootstrap required, opened a new log
I20260923 17:19:51.103145 31801 ts_tablet_manager.cc:1397] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260923 17:19:51.103926 31802 tablet_bootstrap.cc:654] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:51.104720 31802 log.cc:824] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:51.106666 31802 tablet_bootstrap.cc:492] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: No bootstrap required, opened a new log
I20260923 17:19:51.106880 31802 ts_tablet_manager.cc:1397] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:19:51.107152 31800 raft_consensus.cc:348] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.107448 31800 raft_consensus.cc:374] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:51.107537 31800 raft_consensus.cc:729] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb655b09bcae49eaaf4c91acf1506549, State: Initialized, Role: FOLLOWER
I20260923 17:19:51.107555 31801 raft_consensus.cc:348] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.107805 31801 raft_consensus.cc:374] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:51.107854 31801 raft_consensus.cc:729] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37d82581dd14b6db7223f55d05e2da0, State: Initialized, Role: FOLLOWER
I20260923 17:19:51.108076 31800 consensus_queue.cc:260] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.108268 31801 consensus_queue.cc:260] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.109347 31801 ts_tablet_manager.cc:1428] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:19:51.109642 31495 heartbeater.cc:503] Master 127.23.214.190:40645 was elected leader, sending a full tablet report...
I20260923 17:19:51.109715 31769 heartbeater.cc:503] Master 127.23.214.190:40645 was elected leader, sending a full tablet report...
I20260923 17:19:51.111680 31800 ts_tablet_manager.cc:1428] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260923 17:19:51.116556 31802 raft_consensus.cc:348] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.116883 31802 raft_consensus.cc:374] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:51.116966 31802 raft_consensus.cc:729] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b3cb4911834d8287091fdbdac42d2a, State: Initialized, Role: FOLLOWER
I20260923 17:19:51.117408 31802 consensus_queue.cc:260] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.118209 31632 heartbeater.cc:503] Master 127.23.214.190:40645 was elected leader, sending a full tablet report...
I20260923 17:19:51.118539 31802 ts_tablet_manager.cc:1428] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260923 17:19:51.240921 31807 raft_consensus.cc:482] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:51.241187 31807 raft_consensus.cc:504] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.242558 31807 leader_election.cc:288] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811), b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
I20260923 17:19:51.246516 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.242931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:48449 (local address 127.23.214.129:34811)
0923 17:19:51.243060 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.243066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.243085 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:19:51.243220 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.243224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.243277 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.243365 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:51.243689 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.244242 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.245436 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.245854 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.246047 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.246103 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.246291 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.246376 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:19:51.246516 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.242841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:35989 (local address 127.23.214.131:45829)
0923 17:19:51.243117 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.243131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.243230 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.243564 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.243566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.243580 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:51.243707 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.243714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.244458 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.244461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.245306 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.245315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.246048 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.246051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.246189 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.246206 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.246313 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.246383 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":178}
I20260923 17:19:51.246548 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.242938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:45829 (local address 127.23.214.130:35989)
0923 17:19:51.243069 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.243073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.243216 (+ 143us) server_negotiation.cc:407] Connection header received
0923 17:19:51.243403 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.243406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.243455 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.243548 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:51.243825 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.244353 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.245438 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.245840 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.246042 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.246100 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.246352 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.246438 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:19:51.246539 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.242836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:34811 (local address 127.23.214.131:48449)
0923 17:19:51.242961 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:51.242977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:51.243083 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:51.243400 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:51.243404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:51.243424 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:51.243566 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.243573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.244356 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.244360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.245303 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:51.245312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.246026 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.246029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:51.246151 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.246174 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:19:51.246246 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:19:51.246346 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":37}
I20260923 17:19:51.247254 31448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49567b0ed6c24355ac2f8c70b40c4de8" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb655b09bcae49eaaf4c91acf1506549" is_pre_election: true
I20260923 17:19:51.247412 31585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49567b0ed6c24355ac2f8c70b40c4de8" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b3cb4911834d8287091fdbdac42d2a" is_pre_election: true
I20260923 17:19:51.247622 31448 raft_consensus.cc:2454] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 0.
I20260923 17:19:51.247714 31585 raft_consensus.cc:2454] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 0.
I20260923 17:19:51.248275 31655 leader_election.cc:302] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5b3cb4911834d8287091fdbdac42d2a, c37d82581dd14b6db7223f55d05e2da0; no voters:
I20260923 17:19:51.248575 31807 raft_consensus.cc:2781] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:51.248673 31807 raft_consensus.cc:482] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:51.248734 31807 raft_consensus.cc:3037] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:51.249823 31807 raft_consensus.cc:504] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.250284 31807 leader_election.cc:288] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 election: Requested vote from peers cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811), b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
I20260923 17:19:51.250757 31448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49567b0ed6c24355ac2f8c70b40c4de8" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb655b09bcae49eaaf4c91acf1506549"
I20260923 17:19:51.250830 31585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49567b0ed6c24355ac2f8c70b40c4de8" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b3cb4911834d8287091fdbdac42d2a"
I20260923 17:19:51.250949 31448 raft_consensus.cc:3037] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:51.250962 31585 raft_consensus.cc:3037] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:51.252051 31448 raft_consensus.cc:2454] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 1.
I20260923 17:19:51.252074 31585 raft_consensus.cc:2454] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 1.
I20260923 17:19:51.252383 31656 leader_election.cc:302] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c37d82581dd14b6db7223f55d05e2da0, cb655b09bcae49eaaf4c91acf1506549; no voters:
I20260923 17:19:51.252599 31807 raft_consensus.cc:2781] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:51.252882 31807 raft_consensus.cc:686] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 1 LEADER]: Becoming Leader. State: Replica: c37d82581dd14b6db7223f55d05e2da0, State: Running, Role: LEADER
I20260923 17:19:51.253361 31807 consensus_queue.cc:237] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } }
I20260923 17:19:51.257473 30522 catalog_manager.cc:5495] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37d82581dd14b6db7223f55d05e2da0 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "c37d82581dd14b6db7223f55d05e2da0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } health_report { overall_health: UNKNOWN } } }
W20260923 17:19:51.269035 31771 tablet.cc:2369] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:51.273444 31797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.269780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.23.214.131:41939)
0923 17:19:51.269957 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.269961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.269984 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:19:51.270144 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.270147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.270201 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.270294 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:51.270532 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.271031 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.271941 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.272155 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.272235 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.272565 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:51.273043 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:51.273143 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.273205 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.273304 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:19:51.280028 31585 raft_consensus.cc:1261] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Refusing update from remote peer c37d82581dd14b6db7223f55d05e2da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:51.280028 31448 raft_consensus.cc:1261] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Refusing update from remote peer c37d82581dd14b6db7223f55d05e2da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:51.280840 31809 consensus_queue.cc:1035] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:51.281097 31807 consensus_queue.cc:1035] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:51.291471 31814 mvcc.cc:205] Tried to move back new op lower bound from 7332593628274917376 to 7332593628177371136. Current Snapshot: MvccSnapshot[applied={T|T < 7332593628274917376}]
I20260923 17:19:51.290324 31812 mvcc.cc:205] Tried to move back new op lower bound from 7332593628274917376 to 7332593628177371136. Current Snapshot: MvccSnapshot[applied={T|T < 7332593628274917376}]
I20260923 17:19:51.303298 30524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:51.303519 30524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:51.308542 30524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2697f8ca5a64bc591679a6e45f99c91: Sending DeleteTablet for 3 replicas of tablet 49567b0ed6c24355ac2f8c70b40c4de8
I20260923 17:19:51.309572 31428 tablet_service.cc:1514] Processing DeleteTablet for tablet 49567b0ed6c24355ac2f8c70b40c4de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:51 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:45416
I20260923 17:19:51.310041 31565 tablet_service.cc:1514] Processing DeleteTablet for tablet 49567b0ed6c24355ac2f8c70b40c4de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:51 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:51246
I20260923 17:19:51.310248 31822 tablet_replica.cc:330] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: stopping tablet replica
I20260923 17:19:51.310407 31702 tablet_service.cc:1514] Processing DeleteTablet for tablet 49567b0ed6c24355ac2f8c70b40c4de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:51 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:53724
I20260923 17:19:51.310611 31822 raft_consensus.cc:2229] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:51.310927 31824 tablet_replica.cc:330] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: stopping tablet replica
I20260923 17:19:51.310976 31822 raft_consensus.cc:2258] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:51.311262 31824 raft_consensus.cc:2229] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:51.311681 31824 raft_consensus.cc:2258] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:51.312665 31822 ts_tablet_manager.cc:1905] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:51.313071 31824 ts_tablet_manager.cc:1905] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:51.314810 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30492
I20260923 17:19:51.315824 31822 ts_tablet_manager.cc:1918] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:51.315955 31822 log.cc:1196] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/49567b0ed6c24355ac2f8c70b40c4de8
I20260923 17:19:51.316322 31822 ts_tablet_manager.cc:1939] T 49567b0ed6c24355ac2f8c70b40c4de8 P cb655b09bcae49eaaf4c91acf1506549: Deleting consensus metadata
I20260923 17:19:51.317047 31824 ts_tablet_manager.cc:1918] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:51.317200 31824 log.cc:1196] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/49567b0ed6c24355ac2f8c70b40c4de8
I20260923 17:19:51.317579 31824 ts_tablet_manager.cc:1939] T 49567b0ed6c24355ac2f8c70b40c4de8 P c37d82581dd14b6db7223f55d05e2da0: Deleting consensus metadata
I20260923 17:19:51.322443 31823 tablet_replica.cc:330] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: stopping tablet replica
I20260923 17:19:51.322937 31823 raft_consensus.cc:2229] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:51.323457 31823 raft_consensus.cc:2258] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 17:19:51.324157 31381 connection.cc:535] server connection from 127.0.0.1:45416 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 17:19:51.324183 31656 connection.cc:535] server connection from 127.0.0.1:53724 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:19:51.326072 31823 ts_tablet_manager.cc:1905] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:51.329475 31823 ts_tablet_manager.cc:1918] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:51.329650 31823 log.cc:1196] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/49567b0ed6c24355ac2f8c70b40c4de8
I20260923 17:19:51.330135 31823 ts_tablet_manager.cc:1939] T 49567b0ed6c24355ac2f8c70b40c4de8 P b5b3cb4911834d8287091fdbdac42d2a: Deleting consensus metadata
W20260923 17:19:51.331435 31520 connection.cc:441] server connection from 127.0.0.1:51246 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51246 (request call id 1) could send its response
I20260923 17:19:51.332204 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.312895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.23.214.189:43711)
0923 17:19:51.313022 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.313027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.322254 (+ 9227us) server_negotiation.cc:407] Connection header received
0923 17:19:51.322583 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.322587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.322642 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.322697 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.323820 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.324585 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.325735 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.326063 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.326150 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.328170 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.328174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.328176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.328203 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.330049 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.330812 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.330815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.330817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.330900 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.331205 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.331208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.331210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.331431 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.331541 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.331913 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.332041 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:19:51.341130 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.328463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.23.214.189:43711)
0923 17:19:51.328580 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.328585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.329841 (+ 1256us) server_negotiation.cc:407] Connection header received
0923 17:19:51.330103 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.330107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.330158 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.330212 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.331427 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.332306 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.333578 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.334002 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.334079 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.336093 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.336097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.336100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.336126 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.338569 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.339562 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.339566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.339568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.339662 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.339996 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.340000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.340002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.340247 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.340369 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.340788 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.340929 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:19:51.342346 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.329192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.23.214.188:32839)
0923 17:19:51.329315 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.329320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.330697 (+ 1377us) server_negotiation.cc:407] Connection header received
0923 17:19:51.332810 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.332815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.332869 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.332923 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.333862 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.334709 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.335930 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.336258 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.336330 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.338367 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.338370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.338373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.338398 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.340319 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.340916 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.340919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.340921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.341008 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.341322 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.341328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.341333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.341546 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.341643 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.342036 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.342170 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:19:51.343305 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:51.313434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.23.214.188:32839)
0923 17:19:51.313664 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:51.313669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:51.333716 (+ 20047us) server_negotiation.cc:407] Connection header received
0923 17:19:51.333900 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:51.333903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:51.333948 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:51.333998 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:51.335081 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.335816 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.336840 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:51.337054 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:51.337165 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:51.338686 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:51.338690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:51.338692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:51.338714 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:51.341264 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.341990 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.341994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.341995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.342075 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:51.342377 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:51.342382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:51.342387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:51.342576 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:51.342684 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:51.343103 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 17:19:51.343200 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"spinlock_wait_cycles":896}
I20260923 17:19:52.292627 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.292131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:44167)
0923 17:19:52.292439 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.292528 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
W20260923 17:19:52.293150 31495 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:40645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.294198 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.293771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:46233)
0923 17:19:52.294026 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.294103 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":64,"threads_started":1}
W20260923 17:19:52.294482 31632 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:40645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.297449 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.297046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:48877)
0923 17:19:52.297279 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.297352 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":64,"threads_started":1}
W20260923 17:19:52.297850 31769 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:40645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.809540 31836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2697f8ca5a64bc591679a6e45f99c91)
I20260923 17:19:52.809556 31837 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2697f8ca5a64bc591679a6e45f99c91)
I20260923 17:19:52.809702 31837 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:52.809779 31836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:52.810348 31837 leader_election.cc:288] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645), 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189:43711)
I20260923 17:19:52.810927 31836 leader_election.cc:288] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645), f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188:32839)
I20260923 17:19:52.811038 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.810526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38728)
0923 17:19:52.810826 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.810931 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
W20260923 17:19:52.811242 31299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.811717 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.810985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38740)
0923 17:19:52.811544 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.811620 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":66,"threads_started":1}
W20260923 17:19:52.811884 31110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.812366 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.812042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38750)
0923 17:19:52.812252 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.812305 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260923 17:19:52.812515 31299 leader_election.cc:334] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645): Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.812803 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.812566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38766)
0923 17:19:52.812697 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.812745 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 17:19:52.812948 31110 leader_election.cc:334] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645): Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.815686 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.810580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:43711 (local address 127.0.0.1:56592)
0923 17:19:52.811227 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.811245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:52.811347 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:52.811638 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:52.811642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:52.811659 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:52.811774 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:52.811781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.812997 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.813000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:52.814320 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:52.814331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.815257 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.815260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:52.815379 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:52.815403 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:19:52.815467 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 17:19:52.815557 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":37,"thread_start_us":83,"threads_started":1}
I20260923 17:19:52.815706 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.810715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.23.214.189:43711)
0923 17:19:52.811351 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:52.811356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:52.811371 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:19:52.811460 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:52.811465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:52.811516 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:52.811899 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:52.812022 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.812870 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.814473 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.815133 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.815262 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:52.815328 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:52.815510 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 17:19:52.815589 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":51,"threads_started":1}
I20260923 17:19:52.815977 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.811344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.23.214.188:32839)
0923 17:19:52.811566 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:52.811573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:52.811985 (+ 412us) server_negotiation.cc:407] Connection header received
0923 17:19:52.812140 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:52.812143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:52.812195 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:52.812247 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:52.812584 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.813297 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.814835 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.815442 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.815579 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:52.815620 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:52.815824 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 17:19:52.815898 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":152,"thread_start_us":56,"threads_started":1}
I20260923 17:19:52.816249 31130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65c45e932d664f29b5e591afdd6bd804" is_pre_election: true
I20260923 17:19:52.816454 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.811052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:32839 (local address 127.0.0.1:57756)
0923 17:19:52.811918 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.811932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:52.812037 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:52.812366 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:52.812369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:52.812395 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:52.812472 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:52.812477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.813441 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.813444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:52.814617 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:52.814626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:52.815584 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:52.815588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:52.815674 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:52.815699 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:19:52.816166 (+ 467us) client_negotiation.cc:240] Negotiation successful
0923 17:19:52.816239 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":39,"threads_started":1}
I20260923 17:19:52.816468 31130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b1fa64f73f44249b0bd6cd57dd9297 in term 1.
I20260923 17:19:52.816847 31301 leader_election.cc:302] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c45e932d664f29b5e591afdd6bd804, f4b1fa64f73f44249b0bd6cd57dd9297; no voters: e2697f8ca5a64bc591679a6e45f99c91
I20260923 17:19:52.816916 31328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65c45e932d664f29b5e591afdd6bd804" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" is_pre_election: true
I20260923 17:19:52.817044 31837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:19:52.817149 31328 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c45e932d664f29b5e591afdd6bd804 in term 1.
I20260923 17:19:52.817200 31837 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2697f8ca5a64bc591679a6e45f99c91)
I20260923 17:19:52.817236 31837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:19:52.817467 31112 leader_election.cc:302] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c45e932d664f29b5e591afdd6bd804, f4b1fa64f73f44249b0bd6cd57dd9297; no voters: e2697f8ca5a64bc591679a6e45f99c91
I20260923 17:19:52.817685 31836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:19:52.817750 31836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2697f8ca5a64bc591679a6e45f99c91)
I20260923 17:19:52.817804 31836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:19:52.818734 31837 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:52.819070 31836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:52.819164 31837 leader_election.cc:288] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 2 election: Requested vote from peers e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645), 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189:43711)
I20260923 17:19:52.819459 31836 leader_election.cc:288] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [CANDIDATE]: Term 2 election: Requested vote from peers e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645), f4b1fa64f73f44249b0bd6cd57dd9297 (127.23.214.188:32839)
I20260923 17:19:52.819705 31130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65c45e932d664f29b5e591afdd6bd804"
I20260923 17:19:52.819796 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.819537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38782)
0923 17:19:52.819649 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.819732 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:19:52.819787 31328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65c45e932d664f29b5e591afdd6bd804" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297"
I20260923 17:19:52.819923 31130 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f4b1fa64f73f44249b0bd6cd57dd9297 in current term 2: Already voted for candidate 65c45e932d664f29b5e591afdd6bd804 in this term.
I20260923 17:19:52.819942 31328 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c45e932d664f29b5e591afdd6bd804 in current term 2: Already voted for candidate f4b1fa64f73f44249b0bd6cd57dd9297 in this term.
I20260923 17:19:52.820508 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.819319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38780)
0923 17:19:52.820199 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.820252 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 17:19:52.820570 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.820385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38792)
0923 17:19:52.820481 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.820523 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:19:52.820678 31110 leader_election.cc:334] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645): Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.820786 31110 leader_election.cc:302] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c45e932d664f29b5e591afdd6bd804; no voters: e2697f8ca5a64bc591679a6e45f99c91, f4b1fa64f73f44249b0bd6cd57dd9297
I20260923 17:19:52.821036 31836 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 17:19:52.821208 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:52.820999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:38794)
0923 17:19:52.821102 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:52.821152 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:19:52.821308 31299 leader_election.cc:334] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645): Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:52.821372 31299 leader_election.cc:302] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4b1fa64f73f44249b0bd6cd57dd9297; no voters: 65c45e932d664f29b5e591afdd6bd804, e2697f8ca5a64bc591679a6e45f99c91
I20260923 17:19:52.821502 31837 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 17:19:53.294575 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:53.294016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:48201)
0923 17:19:53.294384 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:53.294460 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260923 17:19:53.295471 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:53.295091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:48431)
0923 17:19:53.295337 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:53.295386 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260923 17:19:53.298954 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:53.298467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:47417)
0923 17:19:53.298798 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:53.298874 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":190,"threads_started":1}
I20260923 17:19:54.296079 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.295580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:57591)
0923 17:19:54.295864 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.295950 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260923 17:19:54.296597 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.296136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:41617)
0923 17:19:54.296419 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.296489 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
I20260923 17:19:54.299975 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.299567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:33391)
0923 17:19:54.299840 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.299893 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260923 17:19:54.684332 31852 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:54.684476 31852 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:54.685032 31852 leader_election.cc:288] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645), 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189:43711)
I20260923 17:19:54.685577 31130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65c45e932d664f29b5e591afdd6bd804" is_pre_election: true
I20260923 17:19:54.685702 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.685207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:39048)
0923 17:19:54.685525 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.685599 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
I20260923 17:19:54.685741 31130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b1fa64f73f44249b0bd6cd57dd9297 in term 2.
I20260923 17:19:54.686134 31301 leader_election.cc:302] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c45e932d664f29b5e591afdd6bd804, f4b1fa64f73f44249b0bd6cd57dd9297; no voters:
I20260923 17:19:54.686322 31852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 17:19:54.686400 31852 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:54.686460 31852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:19:54.686698 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.686483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:39062)
0923 17:19:54.686597 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.686644 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 17:19:54.686808 31299 leader_election.cc:334] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645): Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:54.687608 31852 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:54.687940 31852 leader_election.cc:288] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 3 election: Requested vote from peers e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645), 65c45e932d664f29b5e591afdd6bd804 (127.23.214.189:43711)
I20260923 17:19:54.688349 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.688127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:39068)
0923 17:19:54.688211 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.688298 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 17:19:54.688315 31130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65c45e932d664f29b5e591afdd6bd804"
I20260923 17:19:54.688467 31130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:19:54.689011 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.688817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:39084)
0923 17:19:54.688915 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.688965 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 17:19:54.689113 31299 leader_election.cc:334] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645): Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
I20260923 17:19:54.689523 31130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b1fa64f73f44249b0bd6cd57dd9297 in term 3.
I20260923 17:19:54.689838 31301 leader_election.cc:302] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c45e932d664f29b5e591afdd6bd804, f4b1fa64f73f44249b0bd6cd57dd9297; no voters: e2697f8ca5a64bc591679a6e45f99c91
I20260923 17:19:54.690049 31852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:19:54.690268 31852 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [term 3 LEADER]: Becoming Leader. State: Replica: f4b1fa64f73f44249b0bd6cd57dd9297, State: Running, Role: LEADER
I20260923 17:19:54.690541 31852 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } }
I20260923 17:19:54.691963 31855 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b1fa64f73f44249b0bd6cd57dd9297. Latest consensus state: current_term: 3 leader_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:54.692101 31855 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:54.692435 31857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:54.694669 31857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:54.695013 31857 catalog_manager.cc:1259] Loaded cluster ID: 3ad1c6fb21b34c86a4afaefcd9e7447d
I20260923 17:19:54.695057 31857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:54.695552 31857 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:54.695884 31857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Loaded TSK: 0
I20260923 17:19:54.696513 31857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:19:54.736006 31317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:54.739416 31317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:54.742769 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.742433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:39106)
0923 17:19:54.742611 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.742679 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 17:19:54.742792 31130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [term 3 FOLLOWER]: Refusing update from remote peer f4b1fa64f73f44249b0bd6cd57dd9297: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 17:19:54.743355 31855 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 17:19:54.743556 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.743304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.0.0.1:39114)
0923 17:19:54.743419 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.743465 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260923 17:19:54.743724 31299 consensus_peers.cc:487] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 -> Peer e2697f8ca5a64bc591679a6e45f99c91 (127.23.214.190:40645): Couldn't send request to peer e2697f8ca5a64bc591679a6e45f99c91. Status: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:19:54.744902 31859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b1fa64f73f44249b0bd6cd57dd9297. Latest consensus state: current_term: 3 leader_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:54.745042 31859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:19:54.745965 31852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:54.746088 31852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:54.746316 31859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2697f8ca5a64bc591679a6e45f99c91" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40645 } } peers { permanent_uuid: "65c45e932d664f29b5e591afdd6bd804" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 43711 } attrs { promote: false } } peers { permanent_uuid: "f4b1fa64f73f44249b0bd6cd57dd9297" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 32839 } attrs { promote: false } } }
I20260923 17:19:54.746452 31859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c45e932d664f29b5e591afdd6bd804 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:19:54.756780 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.752478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.23.214.129:34811)
0923 17:19:54.752752 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:54.752757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:54.752772 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:19:54.752827 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:54.752829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:54.752878 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:54.752962 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:54.753255 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.754053 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.755601 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.756246 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.756370 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:54.756426 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:54.756578 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 17:19:54.756657 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
I20260923 17:19:54.756780 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.752025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:34811 (local address 127.0.0.1:45418)
0923 17:19:54.752212 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.752227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:54.752319 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:54.752983 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:54.752986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:54.753004 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:54.753131 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:54.753141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.754195 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.754199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:54.755483 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:54.755492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.756368 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.756371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:54.756437 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:54.756458 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:19:54.756556 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:19:54.756630 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":61}
I20260923 17:19:54.757735 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.752885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.23.214.130:35989)
0923 17:19:54.753139 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:54.753143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:54.753359 (+ 216us) server_negotiation.cc:407] Connection header received
0923 17:19:54.753453 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:54.753455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:54.753498 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:54.753588 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:54.753857 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.754615 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.756483 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.757128 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.757258 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:54.757309 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:54.757520 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 17:19:54.757604 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260923 17:19:54.757725 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.752214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:35989 (local address 127.0.0.1:51262)
0923 17:19:54.753259 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.753273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:54.753370 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:54.753645 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:54.753648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:54.753668 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:54.753752 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:54.753756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.754716 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.754722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:54.755713 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:54.755722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.757243 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.757246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:54.757335 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:54.757356 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:19:54.757502 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:19:54.757575 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260923 17:19:54.757756 31428 tablet_service.cc:1467] Processing CreateTablet for tablet 56f230218d9f4a169e43fcd563a10dd4 (DEFAULT_TABLE table=test-table [id=01699ff149c84d74a4674af2dab8f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:54.758152 31428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f230218d9f4a169e43fcd563a10dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:54.758306 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.752558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:41939 (local address 127.0.0.1:53730)
0923 17:19:54.753540 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.753552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:54.753616 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:54.754103 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:54.754106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:54.754124 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:54.754267 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:54.754274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.755835 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.755837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:54.756921 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:54.756930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.757916 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.757921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:54.758010 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:54.758031 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:19:54.758112 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:19:54.758181 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":19,"thread_start_us":93,"threads_started":1}
I20260923 17:19:54.758353 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.752664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.23.214.131:41939)
0923 17:19:54.753811 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:54.753816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:54.753898 (+ 82us) server_negotiation.cc:407] Connection header received
0923 17:19:54.753946 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:54.753949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:54.753994 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:54.754080 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:54.754384 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.755155 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.757065 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.757655 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.757768 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:54.757981 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:54.758143 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 17:19:54.758234 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":61,"threads_started":1}
I20260923 17:19:54.758587 31565 tablet_service.cc:1467] Processing CreateTablet for tablet 56f230218d9f4a169e43fcd563a10dd4 (DEFAULT_TABLE table=test-table [id=01699ff149c84d74a4674af2dab8f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:54.758864 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f230218d9f4a169e43fcd563a10dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:54.759064 31702 tablet_service.cc:1467] Processing CreateTablet for tablet 56f230218d9f4a169e43fcd563a10dd4 (DEFAULT_TABLE table=test-table [id=01699ff149c84d74a4674af2dab8f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:54.759305 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f230218d9f4a169e43fcd563a10dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:54.760898 31870 tablet_bootstrap.cc:492] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549: Bootstrap starting.
I20260923 17:19:54.761963 31871 tablet_bootstrap.cc:492] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Bootstrap starting.
I20260923 17:19:54.762149 31870 tablet_bootstrap.cc:654] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:54.762323 31872 tablet_bootstrap.cc:492] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: Bootstrap starting.
I20260923 17:19:54.762887 31871 tablet_bootstrap.cc:654] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:54.763281 31872 tablet_bootstrap.cc:654] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:54.764499 31870 tablet_bootstrap.cc:492] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549: No bootstrap required, opened a new log
I20260923 17:19:54.764569 31872 tablet_bootstrap.cc:492] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: No bootstrap required, opened a new log
I20260923 17:19:54.764633 31870 ts_tablet_manager.cc:1397] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:19:54.764685 31872 ts_tablet_manager.cc:1397] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:54.765151 31872 raft_consensus.cc:348] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.765146 31870 raft_consensus.cc:348] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.765280 31870 raft_consensus.cc:374] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:54.765278 31872 raft_consensus.cc:374] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:54.765332 31872 raft_consensus.cc:729] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37d82581dd14b6db7223f55d05e2da0, State: Initialized, Role: FOLLOWER
I20260923 17:19:54.765362 31870 raft_consensus.cc:729] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb655b09bcae49eaaf4c91acf1506549, State: Initialized, Role: FOLLOWER
I20260923 17:19:54.765482 31872 consensus_queue.cc:260] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.765528 31870 consensus_queue.cc:260] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.765873 31872 ts_tablet_manager.cc:1428] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:54.765929 31870 ts_tablet_manager.cc:1428] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 17:19:54.766984 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.766420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:43823)
0923 17:19:54.766820 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.766883 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312}
I20260923 17:19:54.767284 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.766678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:36473)
0923 17:19:54.767160 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.767217 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
I20260923 17:19:54.768203 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.767733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:39595)
0923 17:19:54.768095 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.768144 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299}
I20260923 17:19:54.768751 31494 heartbeater.cc:503] Master 127.23.214.188:32839 was elected leader, sending a full tablet report...
I20260923 17:19:54.768790 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.768544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:46847)
0923 17:19:54.768683 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.768727 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 17:19:54.769447 31768 heartbeater.cc:503] Master 127.23.214.188:32839 was elected leader, sending a full tablet report...
I20260923 17:19:54.770324 31871 tablet_bootstrap.cc:492] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: No bootstrap required, opened a new log
I20260923 17:19:54.770538 31871 ts_tablet_manager.cc:1397] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260923 17:19:54.771144 31871 raft_consensus.cc:348] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.771272 31871 raft_consensus.cc:374] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:54.771324 31871 raft_consensus.cc:729] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b3cb4911834d8287091fdbdac42d2a, State: Initialized, Role: FOLLOWER
I20260923 17:19:54.771456 31871 consensus_queue.cc:260] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.771778 31871 ts_tablet_manager.cc:1428] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:54.772361 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.772122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:40261)
0923 17:19:54.772226 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.772287 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 17:19:54.773062 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.772834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:34809)
0923 17:19:54.772956 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.773008 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:19:54.773698 31631 heartbeater.cc:503] Master 127.23.214.188:32839 was elected leader, sending a full tablet report...
W20260923 17:19:54.808133 31634 tablet.cc:2369] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:19:54.849613 31497 tablet.cc:2369] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:54.901340 31876 raft_consensus.cc:482] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:54.901571 31876 raft_consensus.cc:504] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.902057 31876 leader_election.cc:288] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811), b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
I20260923 17:19:54.902578 31585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f230218d9f4a169e43fcd563a10dd4" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b3cb4911834d8287091fdbdac42d2a" is_pre_election: true
I20260923 17:19:54.902599 31448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f230218d9f4a169e43fcd563a10dd4" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb655b09bcae49eaaf4c91acf1506549" is_pre_election: true
I20260923 17:19:54.902758 31585 raft_consensus.cc:2454] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 0.
I20260923 17:19:54.902765 31448 raft_consensus.cc:2454] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 0.
I20260923 17:19:54.903128 31656 leader_election.cc:302] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c37d82581dd14b6db7223f55d05e2da0, cb655b09bcae49eaaf4c91acf1506549; no voters:
I20260923 17:19:54.903326 31876 raft_consensus.cc:2781] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:54.903388 31876 raft_consensus.cc:482] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:54.903431 31876 raft_consensus.cc:3037] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:54.904419 31876 raft_consensus.cc:504] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.904814 31876 leader_election.cc:288] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 election: Requested vote from peers cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811), b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989)
I20260923 17:19:54.905202 31448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f230218d9f4a169e43fcd563a10dd4" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb655b09bcae49eaaf4c91acf1506549"
I20260923 17:19:54.905243 31585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f230218d9f4a169e43fcd563a10dd4" candidate_uuid: "c37d82581dd14b6db7223f55d05e2da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b3cb4911834d8287091fdbdac42d2a"
I20260923 17:19:54.905330 31448 raft_consensus.cc:3037] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:54.905391 31585 raft_consensus.cc:3037] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:54.906450 31448 raft_consensus.cc:2454] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 1.
I20260923 17:19:54.906450 31585 raft_consensus.cc:2454] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37d82581dd14b6db7223f55d05e2da0 in term 1.
I20260923 17:19:54.906783 31656 leader_election.cc:302] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c37d82581dd14b6db7223f55d05e2da0, cb655b09bcae49eaaf4c91acf1506549; no voters:
I20260923 17:19:54.907021 31876 raft_consensus.cc:2781] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:54.907275 31876 raft_consensus.cc:686] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 1 LEADER]: Becoming Leader. State: Replica: c37d82581dd14b6db7223f55d05e2da0, State: Running, Role: LEADER
I20260923 17:19:54.907444 31876 consensus_queue.cc:237] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } }
I20260923 17:19:54.908231 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.907963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:55351)
0923 17:19:54.908083 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.908146 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 17:19:54.909175 31318 catalog_manager.cc:5495] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37d82581dd14b6db7223f55d05e2da0 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "c37d82581dd14b6db7223f55d05e2da0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37d82581dd14b6db7223f55d05e2da0" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 41939 } health_report { overall_health: HEALTHY } } }
I20260923 17:19:54.950613 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.947628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.23.214.131:41939)
0923 17:19:54.947760 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:54.947764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:54.947776 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:19:54.947823 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:54.947825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:54.947866 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:54.947933 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:54.948162 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.948685 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.949565 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:54.949772 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:54.949849 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:54.950181 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:54.950253 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:54.950308 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:54.950454 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 17:19:54.950501 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:19:54.953015 31585 raft_consensus.cc:1261] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Refusing update from remote peer c37d82581dd14b6db7223f55d05e2da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:54.953015 31448 raft_consensus.cc:1261] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Refusing update from remote peer c37d82581dd14b6db7223f55d05e2da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:54.953686 31876 consensus_queue.cc:1035] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb655b09bcae49eaaf4c91acf1506549" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:54.953860 31879 consensus_queue.cc:1035] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b3cb4911834d8287091fdbdac42d2a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:54.955668 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.955358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:57871)
0923 17:19:54.955524 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.955585 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 17:19:54.955973 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.955302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:50691)
0923 17:19:54.955816 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.955895 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
I20260923 17:19:54.959957 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.959614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.130:51641)
0923 17:19:54.959815 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.959873 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260923 17:19:54.959976 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.959631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:35985)
0923 17:19:54.959856 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.959913 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260923 17:19:54.960650 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.960355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.129:54431)
0923 17:19:54.960513 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.960565 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 17:19:54.961836 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.961452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40645 (local address 127.23.214.131:48539)
0923 17:19:54.961706 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.961759 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260923 17:19:54.964721 31316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:54.964923 31316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:54.965292 31883 mvcc.cc:205] Tried to move back new op lower bound from 7332593643322699776 to 7332593643142291456. Current Snapshot: MvccSnapshot[applied={T|T < 7332593643322699776 or (T in {7332593643322699776})}]
I20260923 17:19:54.968940 31316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4b1fa64f73f44249b0bd6cd57dd9297: Sending DeleteTablet for 3 replicas of tablet 56f230218d9f4a169e43fcd563a10dd4
I20260923 17:19:54.969677 31428 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f230218d9f4a169e43fcd563a10dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:54 UTC) from {username='oryx', principal='oryx/127.23.214.188@KRBTEST.COM'} at 127.0.0.1:45418
I20260923 17:19:54.969708 31565 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f230218d9f4a169e43fcd563a10dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:54 UTC) from {username='oryx', principal='oryx/127.23.214.188@KRBTEST.COM'} at 127.0.0.1:51262
I20260923 17:19:54.970140 31891 tablet_replica.cc:330] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549: stopping tablet replica
I20260923 17:19:54.970163 31892 tablet_replica.cc:330] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: stopping tablet replica
I20260923 17:19:54.970278 31892 raft_consensus.cc:2229] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:54.970278 31891 raft_consensus.cc:2229] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:54.970408 31892 raft_consensus.cc:2258] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:54.970435 31891 raft_consensus.cc:2258] T 56f230218d9f4a169e43fcd563a10dd4 P cb655b09bcae49eaaf4c91acf1506549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:54.970950 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 31270
I20260923 17:19:54.971024 31892 ts_tablet_manager.cc:1905] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:54.973260 31892 ts_tablet_manager.cc:1918] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:54.973361 31892 log.cc:1196] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/56f230218d9f4a169e43fcd563a10dd4
I20260923 17:19:54.973624 31892 ts_tablet_manager.cc:1939] T 56f230218d9f4a169e43fcd563a10dd4 P b5b3cb4911834d8287091fdbdac42d2a: Deleting consensus metadata
I20260923 17:19:54.974367 31702 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f230218d9f4a169e43fcd563a10dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:54 UTC) from {username='oryx', principal='oryx/127.23.214.188@KRBTEST.COM'} at 127.0.0.1:53730
I20260923 17:19:54.974437 31299 catalog_manager.cc:4841] TS b5b3cb4911834d8287091fdbdac42d2a (127.23.214.130:35989): tablet 56f230218d9f4a169e43fcd563a10dd4 (table test-table [id=01699ff149c84d74a4674af2dab8f074]) successfully deleted
I20260923 17:19:54.974741 31893 tablet_replica.cc:330] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: stopping tablet replica
I20260923 17:19:54.974893 31893 raft_consensus.cc:2229] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:54.975131 31893 raft_consensus.cc:2258] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:54.975754 31893 ts_tablet_manager.cc:1905] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:54.977855 31893 ts_tablet_manager.cc:1918] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:54.977998 31893 log.cc:1196] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/56f230218d9f4a169e43fcd563a10dd4
I20260923 17:19:54.978300 31893 ts_tablet_manager.cc:1939] T 56f230218d9f4a169e43fcd563a10dd4 P c37d82581dd14b6db7223f55d05e2da0: Deleting consensus metadata
I20260923 17:19:54.979090 31300 catalog_manager.cc:4841] TS c37d82581dd14b6db7223f55d05e2da0 (127.23.214.131:41939): tablet 56f230218d9f4a169e43fcd563a10dd4 (table test-table [id=01699ff149c84d74a4674af2dab8f074]) successfully deleted
I20260923 17:19:54.979249 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 31502
I20260923 17:19:54.980448 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:54.980124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:34811 (local address 127.0.0.1:45428)
0923 17:19:54.980285 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:54.980340 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:34811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 17:19:54.980651 31300 catalog_manager.cc:4584] TS cb655b09bcae49eaaf4c91acf1506549 (127.23.214.129:34811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56f230218d9f4a169e43fcd563a10dd4: Network error: Client connection negotiation failed: client connection to 127.23.214.129:34811: connect: Connection refused (error 111)
I20260923 17:19:54.986235 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 31638
I20260923 17:19:54.992959 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 30988
I20260923 17:19:54.999006 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 31098
2026-09-23T17:19:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7466 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:55 dist-test-slave-9q7f krb5kdc[31896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:55 dist-test-slave-9q7f krb5kdc[31896](info): set up 2 sockets
Sep 23 17:19:55 dist-test-slave-9q7f krb5kdc[31896](info): commencing operation
krb5kdc: starting...
W20260923 17:19:56.057358 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:56 dist-test-slave-9q7f krb5kdc[31896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab.
Entry for principal oryx/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab.
I20260923 17:19:56.190711 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45441
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab
--principal=oryx/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:45441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:56.308228 31909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:56.308694 31909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:56.308848 31909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:56.313356 31909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:56.313436 31909 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:19:56.313455 31909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:56.313475 31909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:56.313493 31909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:56.313549 31909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:56.319468 31909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:45441
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.190.keytab
--principal=oryx/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:56.320678 31909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:56.321781 31909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:56.331140 31915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:56.331189 31909 server_base.cc:1034] running on GCE node
W20260923 17:19:56.331146 31917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:56.331130 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:56.332065 31909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:56.333233 31909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:56.334405 31909 hybrid_clock.cc:648] HybridClock initialized: now 1790183996334386 us; error 31 us; skew 500 ppm
Sep 23 17:19:56 dist-test-slave-9q7f krb5kdc[31896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:56.337218 31909 init.cc:377] Logged in from keytab as oryx/127.23.214.190@KRBTEST.COM (short username oryx)
I20260923 17:19:56.338188 31909 webserver.cc:460] Webserver started at http://127.23.214.190:38961/ using document root <none> and password file <none>
I20260923 17:19:56.338872 31909 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:56.338945 31909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:56.339123 31909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:56.341194 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "764a72f4c61a428e849321a13bca8882"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "0d7518a28902d888cc175e6828e437c6"
server_key_iv: "d6756cb6e99b57ef1eb0af8dbe72d494"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.341724 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "764a72f4c61a428e849321a13bca8882"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "0d7518a28902d888cc175e6828e437c6"
server_key_iv: "d6756cb6e99b57ef1eb0af8dbe72d494"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.345625 31909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:19:56.348220 31924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:56.349798 31909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:56.350018 31909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "764a72f4c61a428e849321a13bca8882"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "0d7518a28902d888cc175e6828e437c6"
server_key_iv: "d6756cb6e99b57ef1eb0af8dbe72d494"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.350169 31909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:56.377410 31909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:56.381100 31909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:56.381386 31909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:56.389665 31909 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:45441
I20260923 17:19:56.389730 31986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:45441 every 8 connection(s)
I20260923 17:19:56.390726 31909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:19:56.393579 31987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:56.398309 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 31909
I20260923 17:19:56.398463 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:19:56.398775 24410 external_mini_cluster.cc:1428] Setting key 275f3288a328f2a2e63d744202ce1dec
I20260923 17:19:56.399895 31987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Bootstrap starting.
I20260923 17:19:56.402343 31987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:56.403345 31987 log.cc:824] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:56.405460 31987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: No bootstrap required, opened a new log
Sep 23 17:19:56 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:56.410126 31987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764a72f4c61a428e849321a13bca8882" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45441 } }
I20260923 17:19:56.410488 31987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:56.410588 31987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764a72f4c61a428e849321a13bca8882, State: Initialized, Role: FOLLOWER
I20260923 17:19:56.411182 31987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764a72f4c61a428e849321a13bca8882" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45441 } }
I20260923 17:19:56.411397 31987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:19:56.411499 31987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:19:56.411618 31987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:56.413017 31987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764a72f4c61a428e849321a13bca8882" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45441 } }
I20260923 17:19:56.413453 31987 leader_election.cc:302] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 764a72f4c61a428e849321a13bca8882; no voters:
I20260923 17:19:56.413888 31987 leader_election.cc:288] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:19:56.413996 31992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:56.414292 31992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [term 1 LEADER]: Becoming Leader. State: Replica: 764a72f4c61a428e849321a13bca8882, State: Running, Role: LEADER
I20260923 17:19:56.414732 31992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764a72f4c61a428e849321a13bca8882" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45441 } }
I20260923 17:19:56.415432 31987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:19:56.416198 31992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764a72f4c61a428e849321a13bca8882. Latest consensus state: current_term: 1 leader_uuid: "764a72f4c61a428e849321a13bca8882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764a72f4c61a428e849321a13bca8882" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45441 } } }
I20260923 17:19:56.416316 31992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:56.416662 31993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "764a72f4c61a428e849321a13bca8882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764a72f4c61a428e849321a13bca8882" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45441 } } }
I20260923 17:19:56.416749 31993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882 [sys.catalog]: This master's current role is: LEADER
I20260923 17:19:56.417099 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.400389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.23.214.190:45441)
0923 17:19:56.400918 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:56.400936 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:56.401004 (+ 68us) server_negotiation.cc:407] Connection header received
0923 17:19:56.401739 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:56.401765 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:56.402252 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:56.402632 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:56.403554 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.404685 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.405814 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.406410 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.406559 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.409593 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:56.409599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:56.409615 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:56.409656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:56.412092 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.412827 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.412836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.412844 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.413045 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.413474 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.413478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.413481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.414206 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:56.415388 (+ 1182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:56.415557 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 17:19:56.415895 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":178,"threads_started":1}
W20260923 17:19:56.420763 32007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:19:56.420862 32007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:19:56.420941 32001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:19:56.421897 32001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:19:56.427660 32001 catalog_manager.cc:1347] Generated new cluster ID: 391fd493bbd545e9ae6d4744439bfa68
I20260923 17:19:56.427748 32001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:19:56.438221 32001 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:19:56.439510 32001 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:19:56.446720 32001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Generated new TSK 0
I20260923 17:19:56.447484 32001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab.
Entry for principal oryx/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab.
I20260923 17:19:56.487473 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45441
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab
--principal=oryx/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:56.609320 32013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:56.609613 32013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:56.609660 32013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:56.614619 32013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:56.614728 32013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:56.614797 32013 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:56.614890 32013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:19:56.620934 32013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab
--principal=oryx/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:45441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:56.622160 32013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:56.623224 32013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:56.633590 32018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:56.633615 32019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:56.633790 32013 server_base.cc:1034] running on GCE node
W20260923 17:19:56.634011 32021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:56.634429 32013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:56.635146 32013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:56.636344 32013 hybrid_clock.cc:648] HybridClock initialized: now 1790183996636334 us; error 31 us; skew 500 ppm
Sep 23 17:19:56 dist-test-slave-9q7f krb5kdc[31896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:56.639430 32013 init.cc:377] Logged in from keytab as oryx/127.23.214.129@KRBTEST.COM (short username oryx)
I20260923 17:19:56.640484 32013 webserver.cc:460] Webserver started at http://127.23.214.129:41273/ using document root <none> and password file <none>
I20260923 17:19:56.641076 32013 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:56.641170 32013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:56.641387 32013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:56.643590 32013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "25544493ec1d4f14bbf1761d6314b411"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "cd65c99acbcced7a83fea541d43a99f9"
server_key_iv: "3e1e8f88c747db9f3a233ca4b9c87936"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.644196 32013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "25544493ec1d4f14bbf1761d6314b411"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "cd65c99acbcced7a83fea541d43a99f9"
server_key_iv: "3e1e8f88c747db9f3a233ca4b9c87936"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.648253 32013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:56.651050 32028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:56.652697 32013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:56.652866 32013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "25544493ec1d4f14bbf1761d6314b411"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "cd65c99acbcced7a83fea541d43a99f9"
server_key_iv: "3e1e8f88c747db9f3a233ca4b9c87936"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.653024 32013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:56.669287 32013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:56.673049 32013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:56.673355 32013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:56.675148 32013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:56.675225 32013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:56.675321 32013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:56.675369 32013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:56 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:56.687908 32013 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:42637
I20260923 17:19:56.688201 32143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:42637 every 8 connection(s)
I20260923 17:19:56.688963 32013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:19:56.691049 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.676079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:33417 (local address 127.23.214.190:45441)
0923 17:19:56.676276 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:56.676284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:56.677329 (+ 1045us) server_negotiation.cc:407] Connection header received
0923 17:19:56.678809 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:56.678815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:56.678898 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:56.679001 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:56.680441 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.680990 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.682362 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.682622 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.682747 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.686915 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:56.686919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:56.686922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:56.686958 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:56.688681 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.689273 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.689277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.689279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.689368 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.689775 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.689780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.689783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.690062 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:56.690175 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:56.690793 (+ 618us) server_negotiation.cc:299] Negotiation successful
0923 17:19:56.690916 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:19:56.691859 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.676578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45441 (local address 127.23.214.129:33417)
0923 17:19:56.677145 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:56.677211 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:56.678552 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:56.679201 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:56.679208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:56.679617 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:56.680276 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.680298 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.681119 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.681122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.682229 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.682246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.682784 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.682788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.683015 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.683701 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:56.683728 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:56.686699 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:56.688818 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.688824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.688839 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.689146 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.689470 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.689473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.689475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.689646 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.690189 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:56.690195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:56.690491 (+ 296us) client_negotiation.cc:769] Sending connection context
0923 17:19:56.690658 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:19:56.690928 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260923 17:19:56.693269 32144 heartbeater.cc:347] Connected to a master server at 127.23.214.190:45441
I20260923 17:19:56.693547 32144 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:56.694224 32144 heartbeater.cc:511] Master 127.23.214.190:45441 requested a full tablet report, sending...
I20260923 17:19:56.695102 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 32013
I20260923 17:19:56.695241 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:19:56.695526 24410 external_mini_cluster.cc:1428] Setting key e74fe3b0e1e6c750a9d48f6bfe10b3d3
I20260923 17:19:56.696046 31941 ts_manager.cc:194] Registered new tserver with Master: 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637)
I20260923 17:19:56.697675 31941 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:33417
I20260923 17:19:56.704092 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.696059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45441 (local address 127.23.214.129:56777)
0923 17:19:56.696433 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:56.696449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:56.696550 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:56.696825 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:56.696830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:56.697073 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:56.697207 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.697214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.698236 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.698239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.699143 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.699156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.699638 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.699642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.699738 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.700352 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:56.700364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:56.700702 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:56.702531 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.702536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.702539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.702746 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.703041 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.703044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.703046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.703136 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.703517 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:56.703520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:56.703624 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 17:19:56.703730 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 17:19:56.703885 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
I20260923 17:19:56.704087 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.696024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:56777 (local address 127.23.214.190:45441)
0923 17:19:56.696144 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:56.696150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:56.696530 (+ 380us) server_negotiation.cc:407] Connection header received
0923 17:19:56.696642 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:56.696645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:56.696687 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:56.696774 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:56.697314 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.698095 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.699287 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.699495 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.699636 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.700855 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:56.700858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:56.700861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:56.700886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:56.702414 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.702848 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.702853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.702855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.702938 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.703233 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.703236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.703238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.703419 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:56.703530 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:56.703804 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 17:19:56.703917 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
WARNING: no policy specified for oryx/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab.
Entry for principal oryx/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab.
I20260923 17:19:56.725858 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45441
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab
--principal=oryx/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:56.841707 32153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:56.842029 32153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:56.842079 32153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:56.846256 32153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:56.846328 32153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:56.846370 32153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:56.846405 32153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:19:56.851941 32153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab
--principal=oryx/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:45441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:56.853044 32153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:56.854219 32153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:19:56.864877 32153 server_base.cc:1034] running on GCE node
W20260923 17:19:56.864899 32161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:56.865109 32158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:56.865144 32159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:56.865576 32153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:56.866389 32153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:56.867589 32153 hybrid_clock.cc:648] HybridClock initialized: now 1790183996867567 us; error 33 us; skew 500 ppm
Sep 23 17:19:56 dist-test-slave-9q7f krb5kdc[31896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:56.870647 32153 init.cc:377] Logged in from keytab as oryx/127.23.214.130@KRBTEST.COM (short username oryx)
I20260923 17:19:56.871634 32153 webserver.cc:460] Webserver started at http://127.23.214.130:46325/ using document root <none> and password file <none>
I20260923 17:19:56.872224 32153 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:56.872359 32153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:56.872601 32153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:56.874867 32153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "8154ded950244256951ee1a4c50bf442"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "815d5537699c0bb7c60dce988dde8843"
server_key_iv: "f565de827b7c8347eda855bdfad0b138"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.875576 32153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "8154ded950244256951ee1a4c50bf442"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "815d5537699c0bb7c60dce988dde8843"
server_key_iv: "f565de827b7c8347eda855bdfad0b138"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.879752 32153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:19:56.882483 32168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:56.883733 32153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:19:56.883898 32153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "8154ded950244256951ee1a4c50bf442"
format_stamp: "Formatted at 2026-09-23 17:19:56 on dist-test-slave-9q7f"
server_key: "815d5537699c0bb7c60dce988dde8843"
server_key_iv: "f565de827b7c8347eda855bdfad0b138"
server_key_version: "encryptionkey@0"
I20260923 17:19:56.884021 32153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:56.901746 32153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:56.905704 32153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:56.906005 32153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:56.907757 32153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:56.907853 32153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:56.907928 32153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:56.907981 32153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:56.919432 32153 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:34991
Sep 23 17:19:56 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:56.919533 32283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:34991 every 8 connection(s)
I20260923 17:19:56.920571 32153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:19:56.923177 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 32153
I20260923 17:19:56.923313 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:19:56.923648 24410 external_mini_cluster.cc:1428] Setting key ab777f1d43b6219dec27e4b2a7f4a269
I20260923 17:19:56.924701 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.908570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:44389 (local address 127.23.214.190:45441)
0923 17:19:56.908750 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:56.908756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:56.909658 (+ 902us) server_negotiation.cc:407] Connection header received
0923 17:19:56.910709 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:56.910713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:56.910773 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:56.910863 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:56.912754 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.913688 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.915779 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.916171 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.916267 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.920358 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:56.920362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:56.920364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:56.920394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:56.922256 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.922807 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.922810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.922811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.922938 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.923398 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.923402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.923403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.923668 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:56.923774 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:56.924462 (+ 688us) server_negotiation.cc:299] Negotiation successful
0923 17:19:56.924580 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:19:56.925736 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.909005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45441 (local address 127.23.214.130:44389)
0923 17:19:56.909476 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:56.909530 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:56.910479 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:56.911113 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:56.911128 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:56.911722 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:56.912521 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.912547 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.913854 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.913858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.915629 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.915641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.916373 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.916377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.916659 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.918228 (+ 1569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:56.918252 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:56.920123 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:56.922411 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.922415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.922428 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.922695 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.923033 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.923036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.923038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.923251 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.923818 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:56.923826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:56.924164 (+ 338us) client_negotiation.cc:769] Sending connection context
0923 17:19:56.924368 (+ 204us) client_negotiation.cc:240] Negotiation successful
0923 17:19:56.924777 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":139,"threads_started":1}
I20260923 17:19:56.927105 32284 heartbeater.cc:347] Connected to a master server at 127.23.214.190:45441
I20260923 17:19:56.927289 32284 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:56.927971 32284 heartbeater.cc:511] Master 127.23.214.190:45441 requested a full tablet report, sending...
I20260923 17:19:56.929206 31940 ts_manager.cc:194] Registered new tserver with Master: 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991)
I20260923 17:19:56.930163 31940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:44389
WARNING: no policy specified for oryx/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.131@KRBTEST.COM" created.
I20260923 17:19:56.938035 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.929509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:42331 (local address 127.23.214.190:45441)
0923 17:19:56.929750 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:56.929755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:56.929806 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:19:56.930013 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:56.930016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:56.930060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:56.930152 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:56.930665 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.931417 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.932699 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.932914 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.932995 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.934383 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:56.934386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:56.934388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:56.934410 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:56.936001 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.936496 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.936498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.936500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.936577 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:56.936876 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:56.936879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:56.936883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:56.937201 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:56.937346 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:56.937696 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 17:19:56.937822 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260923 17:19:56.938217 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:56.929461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45441 (local address 127.23.214.130:42331)
0923 17:19:56.929736 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:56.929751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:56.929866 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:56.930167 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:56.930170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:56.930395 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:56.930557 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.930566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.931567 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.931574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.932562 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:56.932577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:56.933039 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:56.933043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:56.933147 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:56.933675 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:56.933695 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:56.934220 (+ 525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:56.936117 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.936119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.936121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.936391 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.936676 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:56.936678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:56.936679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:56.936762 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:56.937353 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:56.937356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:56.937479 (+ 123us) client_negotiation.cc:769] Sending connection context
0923 17:19:56.937623 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:19:56.937906 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab.
Entry for principal oryx/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab.
I20260923 17:19:56.958742 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:45441
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab
--principal=oryx/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:57.078840 32293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:57.079138 32293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:57.079186 32293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:57.083266 32293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:57.083344 32293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:57.083388 32293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:19:57.083425 32293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:19:57.089123 32293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab
--principal=oryx/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.214.190:45441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:57.090334 32293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:57.091398 32293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:57.101473 32298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:57.101581 32293 server_base.cc:1034] running on GCE node
W20260923 17:19:57.101483 32301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:57.101533 32299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:57.102252 32293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:57.102955 32293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:57.104161 32293 hybrid_clock.cc:648] HybridClock initialized: now 1790183997104148 us; error 35 us; skew 500 ppm
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183997, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:19:57.107304 32293 init.cc:377] Logged in from keytab as oryx/127.23.214.131@KRBTEST.COM (short username oryx)
I20260923 17:19:57.108266 32293 webserver.cc:460] Webserver started at http://127.23.214.131:33923/ using document root <none> and password file <none>
I20260923 17:19:57.108872 32293 fs_manager.cc:359] Metadata directory not provided
I20260923 17:19:57.108973 32293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:19:57.109220 32293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:19:57.111443 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "728cfca7edf047c996bf777cd98e1231"
format_stamp: "Formatted at 2026-09-23 17:19:57 on dist-test-slave-9q7f"
server_key: "e64c35ed391d57f2d02798447646c1b3"
server_key_iv: "1bfc335cd849de03b335c09e8a239eb6"
server_key_version: "encryptionkey@0"
I20260923 17:19:57.112035 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "728cfca7edf047c996bf777cd98e1231"
format_stamp: "Formatted at 2026-09-23 17:19:57 on dist-test-slave-9q7f"
server_key: "e64c35ed391d57f2d02798447646c1b3"
server_key_iv: "1bfc335cd849de03b335c09e8a239eb6"
server_key_version: "encryptionkey@0"
I20260923 17:19:57.116232 32293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:19:57.119159 32308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:57.120529 32293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:57.120685 32293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "728cfca7edf047c996bf777cd98e1231"
format_stamp: "Formatted at 2026-09-23 17:19:57 on dist-test-slave-9q7f"
server_key: "e64c35ed391d57f2d02798447646c1b3"
server_key_iv: "1bfc335cd849de03b335c09e8a239eb6"
server_key_version: "encryptionkey@0"
I20260923 17:19:57.120829 32293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:19:57.132241 32293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:19:57.135790 32293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:19:57.136008 32293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:19:57.137548 32293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:19:57.137624 32293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:19:57.137692 32293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:19:57.137742 32293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183997, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:57.148967 32293 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:37547
I20260923 17:19:57.149294 32423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:37547 every 8 connection(s)
I20260923 17:19:57.150215 32293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:19:57.152307 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.138584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:54891 (local address 127.23.214.190:45441)
0923 17:19:57.138749 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.138755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.139844 (+ 1089us) server_negotiation.cc:407] Connection header received
0923 17:19:57.141224 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.141228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.141286 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.141398 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:57.142810 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.143392 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.144635 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.144925 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.145041 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.147924 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:57.147928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:57.147932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:57.147964 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:57.149933 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.150541 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.150547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.150552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.150651 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.151143 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.151147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.151149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.151384 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:57.151500 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.152072 (+ 572us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.152179 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:19:57.153100 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.139069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45441 (local address 127.23.214.131:54891)
0923 17:19:57.139682 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.139735 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.141002 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.141580 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.141587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.142120 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:57.142657 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.142670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.143530 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.143533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.144498 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.144507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.145104 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.145111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.145409 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.146199 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:57.146216 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:57.147736 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:57.150065 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:57.150070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:57.150083 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:57.150436 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:57.150813 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:57.150819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:57.150824 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:57.151005 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:57.151556 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:57.151563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:57.151851 (+ 288us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.152015 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.152264 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":124,"threads_started":1}
I20260923 17:19:57.154561 32424 heartbeater.cc:347] Connected to a master server at 127.23.214.190:45441
I20260923 17:19:57.154731 32424 heartbeater.cc:464] Registering TS with master...
I20260923 17:19:57.155409 32424 heartbeater.cc:511] Master 127.23.214.190:45441 requested a full tablet report, sending...
I20260923 17:19:57.156662 31940 ts_manager.cc:194] Registered new tserver with Master: 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547)
I20260923 17:19:57.157013 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 32293
I20260923 17:19:57.157125 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:19:57.157444 24410 external_mini_cluster.cc:1428] Setting key cc661fc713377dd8fa0db26e5c6ceb99
I20260923 17:19:57.157488 31940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:54891
I20260923 17:19:57.159477 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:19:57.166628 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.158405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:48239 (local address 127.23.214.190:45441)
0923 17:19:57.158521 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.158525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.158537 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:19:57.158675 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.158678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.158720 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.158798 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:57.159315 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.160028 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.160874 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.161268 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.161395 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.162876 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:57.162882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:57.162888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:57.162919 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:57.164689 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.165202 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.165208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.165211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.165318 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.165695 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.165698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.165699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.165942 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:57.166033 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.166403 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.166503 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:19:57.166884 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.158016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45441 (local address 127.23.214.131:48239)
0923 17:19:57.158456 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.158472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.158563 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.158832 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.158835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.159069 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:57.159183 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.159190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.160157 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.160159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.160752 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.160759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.161429 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.161435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.161560 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.162152 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:57.162172 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:57.162706 (+ 534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:57.164843 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:57.164846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:57.164848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:57.165066 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:57.165451 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:57.165454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:57.165458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:57.165571 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:57.166054 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:57.166061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:57.166195 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.166330 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.166527 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":146,"threads_started":1}
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:19:57.186542 31990 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:57.179889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.23.214.190:45441)
0923 17:19:57.180036 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.180040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.180052 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:19:57.180100 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.180103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.180147 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.180243 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:57.181035 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.181579 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.182530 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.182753 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.182866 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.186398 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56464: BlockingRecv error: recv got EOF from 127.0.0.1:56464 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:57.197055 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.188187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.23.214.190:45441)
0923 17:19:57.188310 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.188313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.188433 (+ 120us) server_negotiation.cc:407] Connection header received
0923 17:19:57.188537 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.188539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.188584 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.188716 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:57.189294 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.189811 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.190843 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.191067 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.191167 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.193701 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:57.193705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:57.193707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:57.193748 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:57.195482 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.195912 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.195915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.195916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.196003 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.196285 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.196288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.196290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.196486 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:57.196600 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.196821 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.196920 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:19:57.200519 31940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:57.203933 31940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:57.220153 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.213566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.23.214.131:37547)
0923 17:19:57.214857 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.214870 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.214892 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:19:57.214970 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.214977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.215188 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.215551 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:57.215829 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.217015 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.218674 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.219377 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.219524 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.219664 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.219864 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.219997 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":106,"threads_started":1}
I20260923 17:19:57.220144 32447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.213356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37547 (local address 127.0.0.1:47728)
0923 17:19:57.213950 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.214007 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.214144 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.215362 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.215364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.215375 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:57.215645 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.215656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.217159 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.217162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.218544 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.218554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.219521 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.219525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.219639 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.219683 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.219816 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.219969 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":167,"thread_start_us":79,"threads_started":1}
I20260923 17:19:57.220427 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.213547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42637 (local address 127.0.0.1:36712)
0923 17:19:57.214056 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.214070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.214144 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.215333 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.215342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.215370 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:57.215633 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.215640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.217159 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.217162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.218926 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.218938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.220032 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.220036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.220122 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.220148 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.220199 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.220282 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":239,"thread_start_us":105,"threads_started":1}
I20260923 17:19:57.220652 32449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.214044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:34991 (local address 127.0.0.1:43184)
0923 17:19:57.214534 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.214548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.214640 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.215724 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.215729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.215753 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:57.215860 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.215865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.217112 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.217117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.218958 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.218967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.220224 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.220227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.220328 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.220351 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.220428 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.220501 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":223,"thread_start_us":109,"threads_started":1}
I20260923 17:19:57.221164 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.214175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.23.214.129:42637)
0923 17:19:57.214582 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.214592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.214617 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:19:57.214748 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.214753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.215118 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.215332 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:57.215777 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.217016 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.219081 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.219890 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.220727 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.220879 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.220932 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.221034 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":86,"threads_started":1}
I20260923 17:19:57.221277 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.214536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.23.214.130:34991)
0923 17:19:57.214834 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.214845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.214871 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:19:57.215154 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.215160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.215353 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.215510 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:57.215983 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.216975 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.219096 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.219888 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.220028 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.220154 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.221024 (+ 870us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.221131 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":123,"threads_started":1}
I20260923 17:19:57.223656 32358 tablet_service.cc:1467] Processing CreateTablet for tablet d7bf28ebc9814462a137ffe5cb825ea3 (DEFAULT_TABLE table=test-table [id=ff03086553d44ddeb0bea7efd78c10a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.224342 32218 tablet_service.cc:1467] Processing CreateTablet for tablet d7bf28ebc9814462a137ffe5cb825ea3 (DEFAULT_TABLE table=test-table [id=ff03086553d44ddeb0bea7efd78c10a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.224339 32078 tablet_service.cc:1467] Processing CreateTablet for tablet d7bf28ebc9814462a137ffe5cb825ea3 (DEFAULT_TABLE table=test-table [id=ff03086553d44ddeb0bea7efd78c10a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.224833 32358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7bf28ebc9814462a137ffe5cb825ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.225471 32218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7bf28ebc9814462a137ffe5cb825ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.225471 32078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7bf28ebc9814462a137ffe5cb825ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.232251 32452 tablet_bootstrap.cc:492] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Bootstrap starting.
I20260923 17:19:57.234227 32453 tablet_bootstrap.cc:492] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Bootstrap starting.
I20260923 17:19:57.234615 32452 tablet_bootstrap.cc:654] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.235462 32452 log.cc:824] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:57.236207 32454 tablet_bootstrap.cc:492] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Bootstrap starting.
I20260923 17:19:57.236523 32453 tablet_bootstrap.cc:654] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.237206 32453 log.cc:824] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:57.238026 32454 tablet_bootstrap.cc:654] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.238660 32454 log.cc:824] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Log is configured to *not* fsync() on all Append() calls
I20260923 17:19:57.239259 32453 tablet_bootstrap.cc:492] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: No bootstrap required, opened a new log
I20260923 17:19:57.239480 32453 ts_tablet_manager.cc:1397] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:19:57.240093 32452 tablet_bootstrap.cc:492] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: No bootstrap required, opened a new log
I20260923 17:19:57.240295 32452 ts_tablet_manager.cc:1397] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260923 17:19:57.243691 32452 raft_consensus.cc:348] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.243747 32453 raft_consensus.cc:348] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.243943 32452 raft_consensus.cc:374] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.243995 32453 raft_consensus.cc:374] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.244041 32452 raft_consensus.cc:729] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.244082 32453 raft_consensus.cc:729] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25544493ec1d4f14bbf1761d6314b411, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.244462 32452 consensus_queue.cc:260] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.244549 32453 consensus_queue.cc:260] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.245105 32424 heartbeater.cc:503] Master 127.23.214.190:45441 was elected leader, sending a full tablet report...
I20260923 17:19:57.245187 32144 heartbeater.cc:503] Master 127.23.214.190:45441 was elected leader, sending a full tablet report...
I20260923 17:19:57.245496 32452 ts_tablet_manager.cc:1428] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:19:57.245543 32453 ts_tablet_manager.cc:1428] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:19:57.247001 32454 tablet_bootstrap.cc:492] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: No bootstrap required, opened a new log
I20260923 17:19:57.247223 32454 ts_tablet_manager.cc:1397] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20260923 17:19:57.250698 32454 raft_consensus.cc:348] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.251050 32454 raft_consensus.cc:374] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.251161 32454 raft_consensus.cc:729] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8154ded950244256951ee1a4c50bf442, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.251683 32454 consensus_queue.cc:260] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.252447 32284 heartbeater.cc:503] Master 127.23.214.190:45441 was elected leader, sending a full tablet report...
I20260923 17:19:57.252722 32454 ts_tablet_manager.cc:1428] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260923 17:19:57.282876 32458 raft_consensus.cc:482] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:57.283177 32458 raft_consensus.cc:504] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.284459 32458 leader_election.cc:288] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991)
I20260923 17:19:57.288516 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.284850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:50839 (local address 127.23.214.130:34991)
0923 17:19:57.284982 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.284987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.285142 (+ 155us) server_negotiation.cc:407] Connection header received
0923 17:19:57.285202 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.285204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.285248 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.285359 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:57.285625 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.286206 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.287377 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.287946 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.288079 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.288136 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.288324 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.288391 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:19:57.288516 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.284772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:34991 (local address 127.23.214.131:50839)
0923 17:19:57.284985 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.285001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.285109 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.285369 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.285372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.285390 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:57.285509 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.285515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.286312 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.286316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.287251 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.287260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.288079 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.288083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.288187 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.288215 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.288302 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.288378 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260923 17:19:57.289062 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.284821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42637 (local address 127.23.214.131:42827)
0923 17:19:57.285188 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.285213 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.285336 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.285600 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.285603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.285618 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:57.285741 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.285748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.286513 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.286516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.287867 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.287877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.288645 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.288648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.288745 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.288771 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.288862 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.288927 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":155,"threads_started":1}
I20260923 17:19:57.289064 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.284945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:42827 (local address 127.23.214.129:42637)
0923 17:19:57.285084 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.285088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.285306 (+ 218us) server_negotiation.cc:407] Connection header received
0923 17:19:57.285449 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.285454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.285502 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.285581 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:57.285863 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.286382 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.288025 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.288516 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.288646 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.288694 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.288880 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.288959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:19:57.289431 32238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bf28ebc9814462a137ffe5cb825ea3" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ded950244256951ee1a4c50bf442" is_pre_election: true
I20260923 17:19:57.289803 32238 raft_consensus.cc:2454] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 0.
I20260923 17:19:57.289827 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bf28ebc9814462a137ffe5cb825ea3" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411" is_pre_election: true
I20260923 17:19:57.290138 32098 raft_consensus.cc:2454] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 0.
I20260923 17:19:57.290396 32309 leader_election.cc:302] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 728cfca7edf047c996bf777cd98e1231, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:57.290721 32458 raft_consensus.cc:2781] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:57.290799 32458 raft_consensus.cc:482] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:57.290843 32458 raft_consensus.cc:3037] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.291970 32458 raft_consensus.cc:504] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.292371 32458 leader_election.cc:288] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 election: Requested vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991)
I20260923 17:19:57.292704 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bf28ebc9814462a137ffe5cb825ea3" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411"
I20260923 17:19:57.292747 32238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bf28ebc9814462a137ffe5cb825ea3" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ded950244256951ee1a4c50bf442"
I20260923 17:19:57.292857 32098 raft_consensus.cc:3037] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.292869 32238 raft_consensus.cc:3037] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.294060 32098 raft_consensus.cc:2454] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 1.
I20260923 17:19:57.294418 32312 leader_election.cc:302] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25544493ec1d4f14bbf1761d6314b411, 728cfca7edf047c996bf777cd98e1231; no voters:
I20260923 17:19:57.294588 32458 raft_consensus.cc:2781] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:57.294626 32238 raft_consensus.cc:2454] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 1.
I20260923 17:19:57.294814 32458 raft_consensus.cc:686] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 1 LEADER]: Becoming Leader. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Running, Role: LEADER
I20260923 17:19:57.295184 32458 consensus_queue.cc:237] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } }
I20260923 17:19:57.298599 31941 catalog_manager.cc:5495] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 reported cstate change: term changed from 0 to 1, leader changed from <none> to 728cfca7edf047c996bf777cd98e1231 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } health_report { overall_health: UNKNOWN } } }
I20260923 17:19:57.319128 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.315347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.23.214.131:37547)
0923 17:19:57.315470 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.315473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.315486 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:19:57.315559 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.315561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.315615 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.315702 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:57.316044 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.316595 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.317551 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.317782 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.317892 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.318251 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:57.318762 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:57.318887 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.318952 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.319006 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:19:57.325739 32238 raft_consensus.cc:1261] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Refusing update from remote peer 728cfca7edf047c996bf777cd98e1231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:57.325739 32098 raft_consensus.cc:1261] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Refusing update from remote peer 728cfca7edf047c996bf777cd98e1231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:57.326507 32458 consensus_queue.cc:1035] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:57.326718 32462 consensus_queue.cc:1035] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:57.337168 32467 mvcc.cc:205] Tried to move back new op lower bound from 7332593653038190592 to 7332593652923797504. Current Snapshot: MvccSnapshot[applied={T|T < 7332593653038190592}]
I20260923 17:19:57.339212 32465 mvcc.cc:205] Tried to move back new op lower bound from 7332593653038190592 to 7332593652923797504. Current Snapshot: MvccSnapshot[applied={T|T < 7332593653038190592 or (T in {7332593653038190592})}]
I20260923 17:19:57.343477 31940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:57.343650 31940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:57.346478 31940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Sending DeleteTablet for 3 replicas of tablet d7bf28ebc9814462a137ffe5cb825ea3
I20260923 17:19:57.347260 32358 tablet_service.cc:1514] Processing DeleteTablet for tablet d7bf28ebc9814462a137ffe5cb825ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:57 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:47728
I20260923 17:19:57.347396 32078 tablet_service.cc:1514] Processing DeleteTablet for tablet d7bf28ebc9814462a137ffe5cb825ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:57 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:36712
I20260923 17:19:57.347483 32218 tablet_service.cc:1514] Processing DeleteTablet for tablet d7bf28ebc9814462a137ffe5cb825ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:57 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:43184
I20260923 17:19:57.347837 32474 tablet_replica.cc:330] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: stopping tablet replica
I20260923 17:19:57.347977 32475 tablet_replica.cc:330] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: stopping tablet replica
I20260923 17:19:57.348208 32474 raft_consensus.cc:2229] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:57.348423 32475 raft_consensus.cc:2229] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:57.348655 32474 raft_consensus.cc:2258] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:57.348769 31940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 17:19:57.348892 32475 raft_consensus.cc:2258] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 17:19:57.349242 31940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:57.350803 32475 ts_tablet_manager.cc:1905] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:57.351328 32474 ts_tablet_manager.cc:1905] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:57.354051 32476 tablet_replica.cc:330] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: stopping tablet replica
I20260923 17:19:57.354303 32474 ts_tablet_manager.cc:1918] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:57.354468 32474 log.cc:1196] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/d7bf28ebc9814462a137ffe5cb825ea3
I20260923 17:19:57.354532 32476 raft_consensus.cc:2229] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:57.354868 32474 ts_tablet_manager.cc:1939] T d7bf28ebc9814462a137ffe5cb825ea3 P 728cfca7edf047c996bf777cd98e1231: Deleting consensus metadata
I20260923 17:19:57.355232 32475 ts_tablet_manager.cc:1918] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:57.355443 32475 log.cc:1196] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/d7bf28ebc9814462a137ffe5cb825ea3
I20260923 17:19:57.355871 32475 ts_tablet_manager.cc:1939] T d7bf28ebc9814462a137ffe5cb825ea3 P 8154ded950244256951ee1a4c50bf442: Deleting consensus metadata
I20260923 17:19:57.356163 32078 tablet_service.cc:1467] Processing CreateTablet for tablet d0ae1564f39a485e910d699fff8b3532 (DEFAULT_TABLE table=test-table [id=c7170d7d2c5e4a8fbda910a6bc7e0da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.356475 32078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ae1564f39a485e910d699fff8b3532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.356580 32476 raft_consensus.cc:2258] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:57.358018 31925 catalog_manager.cc:4841] TS 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991): tablet d7bf28ebc9814462a137ffe5cb825ea3 (table test-table [id=ff03086553d44ddeb0bea7efd78c10a8]) successfully deleted
I20260923 17:19:57.358198 32218 tablet_service.cc:1467] Processing CreateTablet for tablet d0ae1564f39a485e910d699fff8b3532 (DEFAULT_TABLE table=test-table [id=c7170d7d2c5e4a8fbda910a6bc7e0da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.358544 32218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ae1564f39a485e910d699fff8b3532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.358994 32476 ts_tablet_manager.cc:1905] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:57.360396 32454 tablet_bootstrap.cc:492] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Bootstrap starting.
I20260923 17:19:57.360890 31927 catalog_manager.cc:4841] TS 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547): tablet d7bf28ebc9814462a137ffe5cb825ea3 (table test-table [id=ff03086553d44ddeb0bea7efd78c10a8]) successfully deleted
I20260923 17:19:57.361476 32454 tablet_bootstrap.cc:654] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.361503 32358 tablet_service.cc:1467] Processing CreateTablet for tablet d0ae1564f39a485e910d699fff8b3532 (DEFAULT_TABLE table=test-table [id=c7170d7d2c5e4a8fbda910a6bc7e0da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.361771 32476 ts_tablet_manager.cc:1918] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:57.361833 32358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ae1564f39a485e910d699fff8b3532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.361867 32476 log.cc:1196] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/d7bf28ebc9814462a137ffe5cb825ea3
I20260923 17:19:57.362234 32476 ts_tablet_manager.cc:1939] T d7bf28ebc9814462a137ffe5cb825ea3 P 25544493ec1d4f14bbf1761d6314b411: Deleting consensus metadata
I20260923 17:19:57.363585 31928 catalog_manager.cc:4841] TS 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637): tablet d7bf28ebc9814462a137ffe5cb825ea3 (table test-table [id=ff03086553d44ddeb0bea7efd78c10a8]) successfully deleted
I20260923 17:19:57.363750 32452 tablet_bootstrap.cc:492] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Bootstrap starting.
I20260923 17:19:57.364717 32452 tablet_bootstrap.cc:654] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.365517 32453 tablet_bootstrap.cc:492] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Bootstrap starting.
I20260923 17:19:57.366506 32453 tablet_bootstrap.cc:654] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.366724 32454 tablet_bootstrap.cc:492] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: No bootstrap required, opened a new log
I20260923 17:19:57.366832 32454 ts_tablet_manager.cc:1397] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 17:19:57.367344 32454 raft_consensus.cc:348] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.367434 32452 tablet_bootstrap.cc:492] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: No bootstrap required, opened a new log
I20260923 17:19:57.367517 32452 ts_tablet_manager.cc:1397] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:19:57.367556 32454 raft_consensus.cc:374] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.367599 32454 raft_consensus.cc:729] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8154ded950244256951ee1a4c50bf442, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.367794 32454 consensus_queue.cc:260] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.367996 32452 raft_consensus.cc:348] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.368120 32452 raft_consensus.cc:374] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.368160 32452 raft_consensus.cc:729] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.368476 32452 consensus_queue.cc:260] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.368637 32453 tablet_bootstrap.cc:492] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: No bootstrap required, opened a new log
I20260923 17:19:57.368721 32453 ts_tablet_manager.cc:1397] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:57.369165 32453 raft_consensus.cc:348] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.369292 32453 raft_consensus.cc:374] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.369326 32453 raft_consensus.cc:729] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25544493ec1d4f14bbf1761d6314b411, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.369443 32453 consensus_queue.cc:260] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.369742 32453 ts_tablet_manager.cc:1428] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 17:19:57.369828 32454 ts_tablet_manager.cc:1428] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:57.371263 32452 ts_tablet_manager.cc:1428] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20260923 17:19:57.401466 32425 tablet.cc:2369] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:19:57.422219 32285 tablet.cc:2369] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:19:57.441238 32145 tablet.cc:2369] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:19:57.520049 32458 raft_consensus.cc:482] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:57.520248 32458 raft_consensus.cc:504] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.520846 32458 leader_election.cc:288] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991)
I20260923 17:19:57.521332 32238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ae1564f39a485e910d699fff8b3532" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ded950244256951ee1a4c50bf442" is_pre_election: true
I20260923 17:19:57.521319 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ae1564f39a485e910d699fff8b3532" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411" is_pre_election: true
I20260923 17:19:57.521519 32238 raft_consensus.cc:2454] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 0.
I20260923 17:19:57.521522 32098 raft_consensus.cc:2454] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 0.
I20260923 17:19:57.521831 32309 leader_election.cc:302] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 728cfca7edf047c996bf777cd98e1231, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:57.522058 32458 raft_consensus.cc:2781] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:57.522116 32458 raft_consensus.cc:482] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:57.522167 32458 raft_consensus.cc:3037] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.523209 32458 raft_consensus.cc:504] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.523604 32458 leader_election.cc:288] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 election: Requested vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991)
I20260923 17:19:57.524026 32238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ae1564f39a485e910d699fff8b3532" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ded950244256951ee1a4c50bf442"
I20260923 17:19:57.524035 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ae1564f39a485e910d699fff8b3532" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411"
I20260923 17:19:57.524145 32238 raft_consensus.cc:3037] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.524183 32098 raft_consensus.cc:3037] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.525234 32238 raft_consensus.cc:2454] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 1.
I20260923 17:19:57.525358 32098 raft_consensus.cc:2454] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 1.
I20260923 17:19:57.525609 32309 leader_election.cc:302] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 728cfca7edf047c996bf777cd98e1231, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:57.525815 32458 raft_consensus.cc:2781] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:57.525952 32458 raft_consensus.cc:686] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 1 LEADER]: Becoming Leader. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Running, Role: LEADER
I20260923 17:19:57.526160 32458 consensus_queue.cc:237] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.528030 31940 catalog_manager.cc:5495] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 reported cstate change: term changed from 0 to 1, leader changed from <none> to 728cfca7edf047c996bf777cd98e1231 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } health_report { overall_health: HEALTHY } } }
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183996, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.190@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:57.584450 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.575878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45441 (local address 127.0.0.1:56480)
0923 17:19:57.576209 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.576222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.576334 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.576582 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.576584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.577007 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:19:57.577117 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.577123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.577982 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.577985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.578551 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.578558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.579015 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.579017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.579077 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.579614 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:19:57.579627 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:19:57.581084 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:19:57.583011 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:57.583020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:57.583032 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:57.583264 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:57.583515 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:19:57.583517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:19:57.583518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:19:57.583595 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:19:57.583959 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:19:57.583966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:19:57.584083 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.584177 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.584301 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":136,"threads_started":1}
I20260923 17:19:57.584538 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.575921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.23.214.190:45441)
0923 17:19:57.576071 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.576077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.576345 (+ 268us) server_negotiation.cc:407] Connection header received
0923 17:19:57.576422 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.576424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.576466 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.576543 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:57.577225 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.577844 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.578651 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.578903 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.579013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.581332 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:57.581337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:57.581339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:57.581363 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:57.582902 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.583355 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.583358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.583359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.583433 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.583672 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.583674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.583675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.583866 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:57.583941 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.584237 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.584374 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:19:57.588899 31940 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:56480:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:19:57.589586 31940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:57.594626 32078 tablet_service.cc:1467] Processing CreateTablet for tablet 567de97b362d48348866c680d2d502f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f7ed3d6be1d4b279632e183e4fcbc75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:57.594739 32218 tablet_service.cc:1467] Processing CreateTablet for tablet 567de97b362d48348866c680d2d502f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f7ed3d6be1d4b279632e183e4fcbc75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:57.594828 32358 tablet_service.cc:1467] Processing CreateTablet for tablet 567de97b362d48348866c680d2d502f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f7ed3d6be1d4b279632e183e4fcbc75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:19:57.595098 32078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567de97b362d48348866c680d2d502f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.595136 32358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567de97b362d48348866c680d2d502f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.595131 32218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567de97b362d48348866c680d2d502f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.597937 32454 tablet_bootstrap.cc:492] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: Bootstrap starting.
I20260923 17:19:57.598013 32452 tablet_bootstrap.cc:492] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: Bootstrap starting.
I20260923 17:19:57.598660 32453 tablet_bootstrap.cc:492] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: Bootstrap starting.
I20260923 17:19:57.598812 32452 tablet_bootstrap.cc:654] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.598942 32454 tablet_bootstrap.cc:654] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.599677 32453 tablet_bootstrap.cc:654] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.600018 32452 tablet_bootstrap.cc:492] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: No bootstrap required, opened a new log
I20260923 17:19:57.600095 32452 ts_tablet_manager.cc:1397] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:19:57.600359 32454 tablet_bootstrap.cc:492] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: No bootstrap required, opened a new log
I20260923 17:19:57.600453 32454 ts_tablet_manager.cc:1397] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:19:57.600562 32452 raft_consensus.cc:348] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.600711 32452 raft_consensus.cc:374] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.600746 32452 raft_consensus.cc:729] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.600884 32452 consensus_queue.cc:260] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.600935 32453 tablet_bootstrap.cc:492] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: No bootstrap required, opened a new log
I20260923 17:19:57.601022 32453 ts_tablet_manager.cc:1397] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:57.600934 32454 raft_consensus.cc:348] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.601065 32454 raft_consensus.cc:374] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.601096 32454 raft_consensus.cc:729] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8154ded950244256951ee1a4c50bf442, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.601176 32452 ts_tablet_manager.cc:1428] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:57.601230 32454 consensus_queue.cc:260] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.601365 32458 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.601467 32454 ts_tablet_manager.cc:1428] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:57.601495 32458 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:57.601480 32453 raft_consensus.cc:348] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.601603 32453 raft_consensus.cc:374] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.601639 32453 raft_consensus.cc:729] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25544493ec1d4f14bbf1761d6314b411, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.601756 32453 consensus_queue.cc:260] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.602029 32459 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.602162 32459 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:57.602164 32460 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.602298 32460 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:57.602679 32453 ts_tablet_manager.cc:1428] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:19:57.816124 32458 raft_consensus.cc:482] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:57.816298 32458 raft_consensus.cc:504] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.816767 32458 leader_election.cc:288] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991)
I20260923 17:19:57.817284 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567de97b362d48348866c680d2d502f7" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411" is_pre_election: true
I20260923 17:19:57.817278 32238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567de97b362d48348866c680d2d502f7" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ded950244256951ee1a4c50bf442" is_pre_election: true
I20260923 17:19:57.817451 32098 raft_consensus.cc:2454] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 0.
I20260923 17:19:57.817466 32238 raft_consensus.cc:2454] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 0.
I20260923 17:19:57.817826 32309 leader_election.cc:302] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 728cfca7edf047c996bf777cd98e1231, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:57.818061 32458 raft_consensus.cc:2781] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:57.818118 32458 raft_consensus.cc:482] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:57.818145 32458 raft_consensus.cc:3037] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.819137 32458 raft_consensus.cc:504] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.819590 32458 leader_election.cc:288] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 election: Requested vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991)
I20260923 17:19:57.820019 32238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567de97b362d48348866c680d2d502f7" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ded950244256951ee1a4c50bf442"
I20260923 17:19:57.820019 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567de97b362d48348866c680d2d502f7" candidate_uuid: "728cfca7edf047c996bf777cd98e1231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411"
I20260923 17:19:57.820154 32098 raft_consensus.cc:3037] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.820154 32238 raft_consensus.cc:3037] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:57.821290 32238 raft_consensus.cc:2454] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 1.
I20260923 17:19:57.821297 32098 raft_consensus.cc:2454] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728cfca7edf047c996bf777cd98e1231 in term 1.
I20260923 17:19:57.821647 32312 leader_election.cc:302] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25544493ec1d4f14bbf1761d6314b411, 728cfca7edf047c996bf777cd98e1231; no voters:
I20260923 17:19:57.821920 32458 raft_consensus.cc:2781] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:57.822029 32458 raft_consensus.cc:686] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [term 1 LEADER]: Becoming Leader. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Running, Role: LEADER
I20260923 17:19:57.822175 32458 consensus_queue.cc:237] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.822893 32462 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: TxnStatusTablet state changed. Reason: New leader 728cfca7edf047c996bf777cd98e1231. Latest consensus state: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.823002 32462 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:57.823248 32491 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:19:57.823423 32491 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:19:57.824059 31941 catalog_manager.cc:5495] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 reported cstate change: term changed from 0 to 1, leader changed from <none> to 728cfca7edf047c996bf777cd98e1231 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } health_report { overall_health: HEALTHY } } }
I20260923 17:19:57.839915 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.835976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37547 (local address 127.0.0.1:47736)
0923 17:19:57.836176 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.836197 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.836317 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.836697 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.836699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.836706 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:57.836801 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.836805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.837943 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.837946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.838568 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.838577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.839158 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.839160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.839229 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.839279 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:57.839620 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:57.839638 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.839709 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.839799 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:19:57.839942 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.836097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.23.214.131:37547)
0923 17:19:57.836417 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.836422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.836458 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:19:57.836511 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.836514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.836565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.836668 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:57.836907 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.837746 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.838690 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.839044 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.839150 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.839433 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:57.839521 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:57.839592 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.839731 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.839788 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260923 17:19:57.842837 32238 raft_consensus.cc:1261] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Refusing update from remote peer 728cfca7edf047c996bf777cd98e1231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:57.843034 32098 raft_consensus.cc:1261] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Refusing update from remote peer 728cfca7edf047c996bf777cd98e1231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:57.843258 32462 consensus_queue.cc:1035] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:57.843458 32458 consensus_queue.cc:1035] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:57.844741 32460 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: TxnStatusTablet state changed. Reason: New leader 728cfca7edf047c996bf777cd98e1231. Latest consensus state: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.844857 32460 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:57.844795 32459 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: TxnStatusTablet state changed. Reason: New leader 728cfca7edf047c996bf777cd98e1231. Latest consensus state: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.844913 32459 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:57.846361 32458 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.846474 32458 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:57.847683 32460 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.847812 32460 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 8154ded950244256951ee1a4c50bf442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:57.848906 32462 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.849004 32462 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 728cfca7edf047c996bf777cd98e1231: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:19:57.849789 32459 tablet_replica.cc:439] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "728cfca7edf047c996bf777cd98e1231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } } }
I20260923 17:19:57.849928 32459 tablet_replica.cc:442] T 567de97b362d48348866c680d2d502f7 P 25544493ec1d4f14bbf1761d6314b411: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:19:57.865554 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.862014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37547 (local address 127.23.214.131:36643)
0923 17:19:57.862483 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:57.862496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:57.862622 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:57.862838 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:57.862840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:57.862846 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:57.862999 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.863006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.863722 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.863728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.864376 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:57.864384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.864812 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.864816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:57.864925 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.864972 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:57.865253 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:57.865260 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:19:57.865326 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 17:19:57.865401 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":49,"thread_start_us":95,"threads_started":1}
I20260923 17:19:57.865554 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.862039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:36643 (local address 127.23.214.131:37547)
0923 17:19:57.862184 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.862191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.862582 (+ 391us) server_negotiation.cc:407] Connection header received
0923 17:19:57.862704 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.862707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.862754 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.862832 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:57.863108 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.863620 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.864496 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.864710 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.864815 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.865099 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:57.865172 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:57.865252 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.865333 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.865384 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":45}
I20260923 17:19:57.872907 32238 raft_consensus.cc:1261] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Refusing update from remote peer 728cfca7edf047c996bf777cd98e1231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:57.873334 32098 raft_consensus.cc:1261] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Refusing update from remote peer 728cfca7edf047c996bf777cd98e1231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:57.873342 32458 consensus_queue.cc:1035] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:57.873772 32458 consensus_queue.cc:1035] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:57.910084 31939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:57.910283 31939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:57.912324 31939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Sending DeleteTablet for 3 replicas of tablet d0ae1564f39a485e910d699fff8b3532
I20260923 17:19:57.912992 32218 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ae1564f39a485e910d699fff8b3532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:57 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:43184
I20260923 17:19:57.912992 32078 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ae1564f39a485e910d699fff8b3532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:57 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:36712
I20260923 17:19:57.913080 32357 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ae1564f39a485e910d699fff8b3532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:57 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:47728
I20260923 17:19:57.913551 32513 tablet_replica.cc:330] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: stopping tablet replica
I20260923 17:19:57.913614 32512 tablet_replica.cc:330] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: stopping tablet replica
I20260923 17:19:57.913746 32512 raft_consensus.cc:2229] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:57.913745 32513 raft_consensus.cc:2229] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:57.913908 32512 raft_consensus.cc:2258] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:57.914070 32513 raft_consensus.cc:2258] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:57.914633 32512 ts_tablet_manager.cc:1905] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:57.917757 32512 ts_tablet_manager.cc:1918] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:57.917843 32512 log.cc:1196] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/d0ae1564f39a485e910d699fff8b3532
I20260923 17:19:57.917980 32513 ts_tablet_manager.cc:1905] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:57.918129 32512 ts_tablet_manager.cc:1939] T d0ae1564f39a485e910d699fff8b3532 P 25544493ec1d4f14bbf1761d6314b411: Deleting consensus metadata
I20260923 17:19:57.918906 32511 tablet_replica.cc:330] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: stopping tablet replica
I20260923 17:19:57.918960 31928 catalog_manager.cc:4841] TS 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637): tablet d0ae1564f39a485e910d699fff8b3532 (table test-table [id=c7170d7d2c5e4a8fbda910a6bc7e0da3]) successfully deleted
I20260923 17:19:57.919034 32511 raft_consensus.cc:2229] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:57.919149 32511 raft_consensus.cc:2258] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:57.919905 32511 ts_tablet_manager.cc:1905] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:57.920960 32513 ts_tablet_manager.cc:1918] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:57.921065 32513 log.cc:1196] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/d0ae1564f39a485e910d699fff8b3532
I20260923 17:19:57.921358 32513 ts_tablet_manager.cc:1939] T d0ae1564f39a485e910d699fff8b3532 P 728cfca7edf047c996bf777cd98e1231: Deleting consensus metadata
I20260923 17:19:57.922277 31927 catalog_manager.cc:4841] TS 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547): tablet d0ae1564f39a485e910d699fff8b3532 (table test-table [id=c7170d7d2c5e4a8fbda910a6bc7e0da3]) successfully deleted
I20260923 17:19:57.922489 32511 ts_tablet_manager.cc:1918] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:57.922564 32511 log.cc:1196] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/d0ae1564f39a485e910d699fff8b3532
I20260923 17:19:57.922819 32511 ts_tablet_manager.cc:1939] T d0ae1564f39a485e910d699fff8b3532 P 8154ded950244256951ee1a4c50bf442: Deleting consensus metadata
I20260923 17:19:57.923749 31925 catalog_manager.cc:4841] TS 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991): tablet d0ae1564f39a485e910d699fff8b3532 (table test-table [id=c7170d7d2c5e4a8fbda910a6bc7e0da3]) successfully deleted
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:19:57.957465 31990 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:19:57.950016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.23.214.190:45441)
0923 17:19:57.950177 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.950182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.950366 (+ 184us) server_negotiation.cc:407] Connection header received
0923 17:19:57.950561 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.950564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.950612 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.950691 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:57.951601 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.952116 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.953280 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.953497 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.953573 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.957327 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56492: BlockingRecv error: recv got EOF from 127.0.0.1:56492 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 17:19:57 dist-test-slave-9q7f krb5kdc[31896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM
I20260923 17:19:57.968739 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:57.959288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56502 (local address 127.23.214.190:45441)
0923 17:19:57.959416 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:57.959421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:57.959483 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:19:57.959617 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:57.959620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:57.959674 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:57.959766 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:19:57.960333 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.961172 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.962113 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:57.962439 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:57.962557 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:57.964912 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:19:57.964916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:19:57.964919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:19:57.964947 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:19:57.966793 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.967340 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.967346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.967348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.967443 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:19:57.967747 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:19:57.967752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:19:57.967754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:19:57.967973 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:19:57.968122 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:57.968429 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 17:19:57.968557 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:19:57.972012 31939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:57.972572 31939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:57.977538 32218 tablet_service.cc:1467] Processing CreateTablet for tablet 03c0649b22b2416c9ef89f88464306a5 (DEFAULT_TABLE table=test-table [id=4a392c698578411abaacbce68d0ade6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.977581 32078 tablet_service.cc:1467] Processing CreateTablet for tablet 03c0649b22b2416c9ef89f88464306a5 (DEFAULT_TABLE table=test-table [id=4a392c698578411abaacbce68d0ade6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.977917 32218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c0649b22b2416c9ef89f88464306a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.977941 32078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c0649b22b2416c9ef89f88464306a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.977916 32357 tablet_service.cc:1467] Processing CreateTablet for tablet 03c0649b22b2416c9ef89f88464306a5 (DEFAULT_TABLE table=test-table [id=4a392c698578411abaacbce68d0ade6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:57.978194 32357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c0649b22b2416c9ef89f88464306a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:57.980480 32454 tablet_bootstrap.cc:492] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: Bootstrap starting.
I20260923 17:19:57.980667 32452 tablet_bootstrap.cc:492] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: Bootstrap starting.
I20260923 17:19:57.980772 32453 tablet_bootstrap.cc:492] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: Bootstrap starting.
I20260923 17:19:57.981310 32454 tablet_bootstrap.cc:654] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.981604 32453 tablet_bootstrap.cc:654] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.981604 32452 tablet_bootstrap.cc:654] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:57.983224 32453 tablet_bootstrap.cc:492] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: No bootstrap required, opened a new log
I20260923 17:19:57.983351 32453 ts_tablet_manager.cc:1397] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:57.983415 32454 tablet_bootstrap.cc:492] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: No bootstrap required, opened a new log
I20260923 17:19:57.983505 32454 ts_tablet_manager.cc:1397] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 17:19:57.983850 32453 raft_consensus.cc:348] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.983982 32453 raft_consensus.cc:374] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.983951 32454 raft_consensus.cc:348] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.984027 32453 raft_consensus.cc:729] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25544493ec1d4f14bbf1761d6314b411, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.984055 32454 raft_consensus.cc:374] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.984094 32454 raft_consensus.cc:729] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8154ded950244256951ee1a4c50bf442, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.984165 32453 consensus_queue.cc:260] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.984207 32454 consensus_queue.cc:260] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.984455 32453 ts_tablet_manager.cc:1428] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:57.984460 32454 ts_tablet_manager.cc:1428] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:57.985513 32452 tablet_bootstrap.cc:492] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: No bootstrap required, opened a new log
I20260923 17:19:57.985620 32452 ts_tablet_manager.cc:1397] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 17:19:57.986135 32452 raft_consensus.cc:348] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.986263 32452 raft_consensus.cc:374] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:57.986317 32452 raft_consensus.cc:729] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Initialized, Role: FOLLOWER
I20260923 17:19:57.986455 32452 consensus_queue.cc:260] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:57.986792 32452 ts_tablet_manager.cc:1428] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:58.009230 32460 raft_consensus.cc:482] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:58.009384 32460 raft_consensus.cc:504] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.010841 32460 leader_election.cc:288] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547)
I20260923 17:19:58.015487 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.011019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42637 (local address 127.23.214.130:57781)
0923 17:19:58.011360 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:58.011387 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:58.011496 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:58.011811 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:58.011815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:58.011836 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:58.011990 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.011998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.013073 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.013077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.014175 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.014183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.015024 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.015027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.015131 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.015158 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:19:58.015236 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:19:58.015306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":80,"threads_started":1}
I20260923 17:19:58.015492 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.011133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:57781 (local address 127.23.214.129:42637)
0923 17:19:58.011471 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:58.011481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:58.011499 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:19:58.011587 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:58.011593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:58.011651 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:58.011737 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:58.012105 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.012917 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.014288 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.014887 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.015007 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.015050 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:58.015269 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 17:19:58.015326 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20260923 17:19:58.016014 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.011250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:60591 (local address 127.23.214.131:37547)
0923 17:19:58.011379 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:58.011385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:58.011645 (+ 260us) server_negotiation.cc:407] Connection header received
0923 17:19:58.011813 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:58.011817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:58.011882 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:58.011981 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:19:58.012765 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.013524 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.014772 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.015369 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.015485 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.015545 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:58.015782 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 17:19:58.015878 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:19:58.015976 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c0649b22b2416c9ef89f88464306a5" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411" is_pre_election: true
I20260923 17:19:58.016157 32098 raft_consensus.cc:2454] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 0.
I20260923 17:19:58.016446 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.011132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37547 (local address 127.23.214.130:60591)
0923 17:19:58.011534 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:58.011550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:58.011651 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:58.011999 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:58.012001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:58.012016 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:19:58.012611 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.012621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.013641 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.013644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.014632 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.014642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.015493 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.015496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.015600 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.015632 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:19:58.016179 (+ 547us) client_negotiation.cc:240] Negotiation successful
0923 17:19:58.016257 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":111,"threads_started":1}
I20260923 17:19:58.016526 32172 leader_election.cc:302] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25544493ec1d4f14bbf1761d6314b411, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:58.016892 32460 raft_consensus.cc:2781] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:58.016968 32460 raft_consensus.cc:482] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:58.016930 32378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c0649b22b2416c9ef89f88464306a5" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728cfca7edf047c996bf777cd98e1231" is_pre_election: true
I20260923 17:19:58.017017 32460 raft_consensus.cc:3037] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:58.017184 32378 raft_consensus.cc:2454] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 0.
I20260923 17:19:58.018262 32460 raft_consensus.cc:504] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.018630 32460 leader_election.cc:288] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 election: Requested vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547)
I20260923 17:19:58.019004 32378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c0649b22b2416c9ef89f88464306a5" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728cfca7edf047c996bf777cd98e1231"
I20260923 17:19:58.018996 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c0649b22b2416c9ef89f88464306a5" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411"
I20260923 17:19:58.019112 32378 raft_consensus.cc:3037] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:58.019137 32098 raft_consensus.cc:3037] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:58.019980 32098 raft_consensus.cc:2454] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 1.
I20260923 17:19:58.020129 32378 raft_consensus.cc:2454] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 1.
I20260923 17:19:58.020274 32172 leader_election.cc:302] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25544493ec1d4f14bbf1761d6314b411, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:58.020481 32460 raft_consensus.cc:2781] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:58.020721 32460 raft_consensus.cc:686] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 1 LEADER]: Becoming Leader. State: Replica: 8154ded950244256951ee1a4c50bf442, State: Running, Role: LEADER
I20260923 17:19:58.020983 32460 consensus_queue.cc:237] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.023085 31940 catalog_manager.cc:5495] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8154ded950244256951ee1a4c50bf442 (127.23.214.130). New cstate: current_term: 1 leader_uuid: "8154ded950244256951ee1a4c50bf442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } health_report { overall_health: UNKNOWN } } }
I20260923 17:19:58.033550 32535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.029317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.23.214.130:34991)
0923 17:19:58.029639 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:58.029646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:58.029658 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:19:58.029729 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:58.029732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:58.029790 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:58.029990 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:58.030271 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.030861 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.031952 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.032224 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.032346 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.032632 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:58.033118 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:58.033269 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:58.033308 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 17:19:58.033365 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260923 17:19:58.036342 32098 raft_consensus.cc:1261] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Refusing update from remote peer 8154ded950244256951ee1a4c50bf442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:58.036489 32378 raft_consensus.cc:1261] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Refusing update from remote peer 8154ded950244256951ee1a4c50bf442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:58.036949 32532 consensus_queue.cc:1035] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:58.037127 32460 consensus_queue.cc:1035] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:58.049286 31941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:58.049468 31941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:58.051602 31941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Sending DeleteTablet for 3 replicas of tablet 03c0649b22b2416c9ef89f88464306a5
I20260923 17:19:58.052381 32078 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c0649b22b2416c9ef89f88464306a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:58 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:36712
I20260923 17:19:58.052511 32218 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c0649b22b2416c9ef89f88464306a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:58 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:43184
I20260923 17:19:58.052589 32357 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c0649b22b2416c9ef89f88464306a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:58 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:47728
I20260923 17:19:58.052778 32512 tablet_replica.cc:330] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: stopping tablet replica
I20260923 17:19:58.052907 32512 raft_consensus.cc:2229] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:58.053040 31941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:19:58.053458 31941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:19:58.053577 32512 raft_consensus.cc:2258] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:58.053813 32511 tablet_replica.cc:330] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: stopping tablet replica
I20260923 17:19:58.054003 32511 raft_consensus.cc:2229] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:58.054363 32511 raft_consensus.cc:2258] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:58.054559 32513 tablet_replica.cc:330] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: stopping tablet replica
I20260923 17:19:58.054718 32513 raft_consensus.cc:2229] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:58.054868 32513 raft_consensus.cc:2258] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:58.055696 32511 ts_tablet_manager.cc:1905] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:58.055891 32513 ts_tablet_manager.cc:1905] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:58.058696 32512 ts_tablet_manager.cc:1905] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:58.059635 32511 ts_tablet_manager.cc:1918] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:58.059710 32511 log.cc:1196] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/03c0649b22b2416c9ef89f88464306a5
I20260923 17:19:58.059964 32511 ts_tablet_manager.cc:1939] T 03c0649b22b2416c9ef89f88464306a5 P 8154ded950244256951ee1a4c50bf442: Deleting consensus metadata
I20260923 17:19:58.060873 32078 tablet_service.cc:1467] Processing CreateTablet for tablet fb191a0e166241cd8e9ffd5846924174 (DEFAULT_TABLE table=test-table [id=1d16f29839994e53b1c921921434de03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:58.061134 32078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb191a0e166241cd8e9ffd5846924174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:58.061347 31925 catalog_manager.cc:4841] TS 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991): tablet 03c0649b22b2416c9ef89f88464306a5 (table test-table [id=4a392c698578411abaacbce68d0ade6c]) successfully deleted
I20260923 17:19:58.061617 32218 tablet_service.cc:1467] Processing CreateTablet for tablet fb191a0e166241cd8e9ffd5846924174 (DEFAULT_TABLE table=test-table [id=1d16f29839994e53b1c921921434de03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:58.061885 32218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb191a0e166241cd8e9ffd5846924174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:58.063126 32453 tablet_bootstrap.cc:492] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411: Bootstrap starting.
I20260923 17:19:58.063359 32357 tablet_service.cc:1467] Processing CreateTablet for tablet fb191a0e166241cd8e9ffd5846924174 (DEFAULT_TABLE table=test-table [id=1d16f29839994e53b1c921921434de03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:19:58.064177 32454 tablet_bootstrap.cc:492] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: Bootstrap starting.
I20260923 17:19:58.064180 32453 tablet_bootstrap.cc:654] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:58.065062 32454 tablet_bootstrap.cc:654] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:58.065378 32453 tablet_bootstrap.cc:492] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411: No bootstrap required, opened a new log
I20260923 17:19:58.065393 32513 ts_tablet_manager.cc:1918] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:58.065465 32513 log.cc:1196] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/03c0649b22b2416c9ef89f88464306a5
I20260923 17:19:58.065464 32453 ts_tablet_manager.cc:1397] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:58.065747 32513 ts_tablet_manager.cc:1939] T 03c0649b22b2416c9ef89f88464306a5 P 728cfca7edf047c996bf777cd98e1231: Deleting consensus metadata
I20260923 17:19:58.066041 32453 raft_consensus.cc:348] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.066166 32453 raft_consensus.cc:374] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:58.066202 32453 raft_consensus.cc:729] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25544493ec1d4f14bbf1761d6314b411, State: Initialized, Role: FOLLOWER
I20260923 17:19:58.066341 32453 consensus_queue.cc:260] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.066489 31927 catalog_manager.cc:4841] TS 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547): tablet 03c0649b22b2416c9ef89f88464306a5 (table test-table [id=4a392c698578411abaacbce68d0ade6c]) successfully deleted
I20260923 17:19:58.066681 32453 ts_tablet_manager.cc:1428] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 17:19:58.066697 32357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb191a0e166241cd8e9ffd5846924174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:19:58.068971 32452 tablet_bootstrap.cc:492] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: Bootstrap starting.
I20260923 17:19:58.069896 32452 tablet_bootstrap.cc:654] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: Neither blocks nor log segments found. Creating new log.
I20260923 17:19:58.070447 32512 ts_tablet_manager.cc:1918] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:19:58.070576 32512 log.cc:1196] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/03c0649b22b2416c9ef89f88464306a5
I20260923 17:19:58.070858 32454 tablet_bootstrap.cc:492] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: No bootstrap required, opened a new log
I20260923 17:19:58.070880 32512 ts_tablet_manager.cc:1939] T 03c0649b22b2416c9ef89f88464306a5 P 25544493ec1d4f14bbf1761d6314b411: Deleting consensus metadata
I20260923 17:19:58.070941 32454 ts_tablet_manager.cc:1397] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260923 17:19:58.071125 32452 tablet_bootstrap.cc:492] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: No bootstrap required, opened a new log
I20260923 17:19:58.071201 32452 ts_tablet_manager.cc:1397] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:19:58.071400 32454 raft_consensus.cc:348] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.071528 32454 raft_consensus.cc:374] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:58.071559 32454 raft_consensus.cc:729] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8154ded950244256951ee1a4c50bf442, State: Initialized, Role: FOLLOWER
I20260923 17:19:58.071570 31928 catalog_manager.cc:4841] TS 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637): tablet 03c0649b22b2416c9ef89f88464306a5 (table test-table [id=4a392c698578411abaacbce68d0ade6c]) successfully deleted
I20260923 17:19:58.071633 32452 raft_consensus.cc:348] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.071671 32454 consensus_queue.cc:260] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.071739 32452 raft_consensus.cc:374] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:19:58.071775 32452 raft_consensus.cc:729] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728cfca7edf047c996bf777cd98e1231, State: Initialized, Role: FOLLOWER
I20260923 17:19:58.071918 32454 ts_tablet_manager.cc:1428] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:58.071898 32452 consensus_queue.cc:260] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.072173 32452 ts_tablet_manager.cc:1428] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:19:58.188982 32460 raft_consensus.cc:482] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:19:58.189145 32460 raft_consensus.cc:504] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.189678 32460 leader_election.cc:288] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547)
I20260923 17:19:58.190176 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb191a0e166241cd8e9ffd5846924174" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411" is_pre_election: true
I20260923 17:19:58.190336 32098 raft_consensus.cc:2454] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 0.
I20260923 17:19:58.190305 32378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb191a0e166241cd8e9ffd5846924174" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728cfca7edf047c996bf777cd98e1231" is_pre_election: true
I20260923 17:19:58.190433 32378 raft_consensus.cc:2454] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 0.
I20260923 17:19:58.190702 32172 leader_election.cc:302] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25544493ec1d4f14bbf1761d6314b411, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:58.190901 32460 raft_consensus.cc:2781] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:19:58.190979 32460 raft_consensus.cc:482] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:19:58.191033 32460 raft_consensus.cc:3037] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:58.192065 32460 raft_consensus.cc:504] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.192442 32460 leader_election.cc:288] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 election: Requested vote from peers 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637), 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547)
I20260923 17:19:58.192836 32098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb191a0e166241cd8e9ffd5846924174" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25544493ec1d4f14bbf1761d6314b411"
I20260923 17:19:58.192870 32378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb191a0e166241cd8e9ffd5846924174" candidate_uuid: "8154ded950244256951ee1a4c50bf442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728cfca7edf047c996bf777cd98e1231"
I20260923 17:19:58.192958 32098 raft_consensus.cc:3037] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:58.193003 32378 raft_consensus.cc:3037] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:19:58.194042 32098 raft_consensus.cc:2454] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 1.
I20260923 17:19:58.194202 32378 raft_consensus.cc:2454] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8154ded950244256951ee1a4c50bf442 in term 1.
I20260923 17:19:58.194432 32172 leader_election.cc:302] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25544493ec1d4f14bbf1761d6314b411, 8154ded950244256951ee1a4c50bf442; no voters:
I20260923 17:19:58.194690 32460 raft_consensus.cc:2781] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:19:58.194799 32460 raft_consensus.cc:686] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 1 LEADER]: Becoming Leader. State: Replica: 8154ded950244256951ee1a4c50bf442, State: Running, Role: LEADER
I20260923 17:19:58.194944 32460 consensus_queue.cc:237] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } }
I20260923 17:19:58.196640 31939 catalog_manager.cc:5495] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8154ded950244256951ee1a4c50bf442 (127.23.214.130). New cstate: current_term: 1 leader_uuid: "8154ded950244256951ee1a4c50bf442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8154ded950244256951ee1a4c50bf442" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 } health_report { overall_health: UNKNOWN } } }
I20260923 17:19:58.239954 32546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.236361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37547 (local address 127.23.214.130:52577)
0923 17:19:58.236726 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:58.236740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:58.236867 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:58.237130 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:58.237133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:58.237141 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:58.237261 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.237271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.238054 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.238058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.238735 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.238744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.239204 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.239207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.239300 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.239349 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:58.239654 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:58.239662 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:19:58.239722 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 17:19:58.239815 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260923 17:19:58.239948 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.236507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:52577 (local address 127.23.214.131:37547)
0923 17:19:58.236653 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:58.236658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:58.236813 (+ 155us) server_negotiation.cc:407] Connection header received
0923 17:19:58.236981 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:58.236984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:58.237036 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:58.237119 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:58.237375 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.237934 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.238865 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.239089 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.239199 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.239493 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:58.239564 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:58.239648 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:58.239742 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 17:19:58.239804 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:19:58.245571 32378 raft_consensus.cc:1261] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Refusing update from remote peer 8154ded950244256951ee1a4c50bf442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:58.245571 32098 raft_consensus.cc:1261] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Refusing update from remote peer 8154ded950244256951ee1a4c50bf442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:19:58.246136 32538 consensus_queue.cc:1035] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "728cfca7edf047c996bf777cd98e1231" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:58.246372 32460 consensus_queue.cc:1035] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25544493ec1d4f14bbf1761d6314b411" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:19:58.251701 32497 mvcc.cc:205] Tried to move back new op lower bound from 7332593656810668032 to 7332593656607764480. Current Snapshot: MvccSnapshot[applied={T|T < 7332593656810668032}]
I20260923 17:19:58.263983 32535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.260499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:56639 (local address 127.23.214.130:34991)
0923 17:19:58.260619 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:19:58.260622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:19:58.260637 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:19:58.260790 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:19:58.260793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:19:58.260851 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:19:58.260935 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:19:58.261165 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.261675 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.262725 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.263026 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.263161 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.263488 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:19:58.263577 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:19:58.263654 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:19:58.263792 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:19:58.263854 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:19:58.264002 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.260457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:34991 (local address 127.23.214.131:56639)
0923 17:19:58.260589 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:58.260607 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:19:58.260681 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:19:58.260965 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:19:58.260968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:19:58.260975 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:19:58.261086 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.261091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.261772 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.261825 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.262589 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:19:58.262601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:19:58.263162 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:19:58.263165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:19:58.263259 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:19:58.263292 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:19:58.263683 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:19:58.263693 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 17:19:58.263778 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:19:58.263865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 17:19:58.284444 31939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:58.284637 31939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:19:58.286948 31939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 764a72f4c61a428e849321a13bca8882: Sending DeleteTablet for 3 replicas of tablet fb191a0e166241cd8e9ffd5846924174
I20260923 17:19:58.287636 32218 tablet_service.cc:1514] Processing DeleteTablet for tablet fb191a0e166241cd8e9ffd5846924174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:58 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:43184
I20260923 17:19:58.287663 32357 tablet_service.cc:1514] Processing DeleteTablet for tablet fb191a0e166241cd8e9ffd5846924174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:58 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:47728
I20260923 17:19:58.287684 32078 tablet_service.cc:1514] Processing DeleteTablet for tablet fb191a0e166241cd8e9ffd5846924174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:19:58 UTC) from {username='oryx', principal='oryx/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:36712
I20260923 17:19:58.287839 32513 tablet_replica.cc:330] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: stopping tablet replica
I20260923 17:19:58.287839 32511 tablet_replica.cc:330] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: stopping tablet replica
I20260923 17:19:58.287947 32511 raft_consensus.cc:2229] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:19:58.287954 32513 raft_consensus.cc:2229] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:58.288072 32513 raft_consensus.cc:2258] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:58.288108 32512 tablet_replica.cc:330] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411: stopping tablet replica
I20260923 17:19:58.288137 32511 raft_consensus.cc:2258] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:58.288225 32512 raft_consensus.cc:2229] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:19:58.288336 32512 raft_consensus.cc:2258] T fb191a0e166241cd8e9ffd5846924174 P 25544493ec1d4f14bbf1761d6314b411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:19:58.288498 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 32013
I20260923 17:19:58.289288 32513 ts_tablet_manager.cc:1905] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:58.290472 32511 ts_tablet_manager.cc:1905] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:19:58.292299 32513 ts_tablet_manager.cc:1918] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:58.292399 32513 log.cc:1196] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/fb191a0e166241cd8e9ffd5846924174
I20260923 17:19:58.292647 32513 ts_tablet_manager.cc:1939] T fb191a0e166241cd8e9ffd5846924174 P 728cfca7edf047c996bf777cd98e1231: Deleting consensus metadata
I20260923 17:19:58.293467 31927 catalog_manager.cc:4841] TS 728cfca7edf047c996bf777cd98e1231 (127.23.214.131:37547): tablet fb191a0e166241cd8e9ffd5846924174 (table test-table [id=1d16f29839994e53b1c921921434de03]) successfully deleted
I20260923 17:19:58.293535 32511 ts_tablet_manager.cc:1918] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:19:58.293656 32511 log.cc:1196] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/fb191a0e166241cd8e9ffd5846924174
I20260923 17:19:58.293977 32511 ts_tablet_manager.cc:1939] T fb191a0e166241cd8e9ffd5846924174 P 8154ded950244256951ee1a4c50bf442: Deleting consensus metadata
I20260923 17:19:58.294740 31925 catalog_manager.cc:4841] TS 8154ded950244256951ee1a4c50bf442 (127.23.214.130:34991): tablet fb191a0e166241cd8e9ffd5846924174 (table test-table [id=1d16f29839994e53b1c921921434de03]) successfully deleted
I20260923 17:19:58.297138 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 32153
W20260923 17:19:58.297163 31928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:42637 (error 108)
I20260923 17:19:58.298681 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:19:58.298265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42637 (local address 127.0.0.1:36720)
0923 17:19:58.298496 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 17:19:58.298584 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
W20260923 17:19:58.298852 31928 catalog_manager.cc:4584] TS 25544493ec1d4f14bbf1761d6314b411 (127.23.214.129:42637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb191a0e166241cd8e9ffd5846924174: Network error: Client connection negotiation failed: client connection to 127.23.214.129:42637: connect: Connection refused (error 111)
I20260923 17:19:58.304999 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 32293
I20260923 17:19:58.312048 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 31909
2026-09-23T17:19:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3311 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:58 dist-test-slave-9q7f krb5kdc[32554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:58 dist-test-slave-9q7f krb5kdc[32554](info): set up 2 sockets
Sep 23 17:19:58 dist-test-slave-9q7f krb5kdc[32554](info): commencing operation
krb5kdc: starting...
W20260923 17:19:59.384672 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:19:59 dist-test-slave-9q7f krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790183999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:19:59.513190 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42105
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:42105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:19:59.629536 32567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:19:59.629868 32567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:19:59.629967 32567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:19:59.634255 32567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:19:59.634351 32567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:19:59.634388 32567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:19:59.634426 32567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:19:59.634460 32567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:19:59.640101 32567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:42105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:19:59.641314 32567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:19:59.642498 32567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:19:59.651768 32575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:59.651758 32572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:19:59.651758 32573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:19:59.652223 32567 server_base.cc:1034] running on GCE node
I20260923 17:19:59.652832 32567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:19:59.654126 32567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:19:59.655301 32567 hybrid_clock.cc:648] HybridClock initialized: now 1790183999655266 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T17:19:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1346 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:19:59 dist-test-slave-9q7f krb5kdc[32580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:19:59 dist-test-slave-9q7f krb5kdc[32580](info): set up 2 sockets
Sep 23 17:19:59 dist-test-slave-9q7f krb5kdc[32580](info): commencing operation
krb5kdc: starting...
W20260923 17:20:00.712922 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:20:00 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:20:00.846289 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:39867
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:39867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:00.963286 32593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:00.963639 32593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:00.963726 32593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:00.968112 32593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:20:00.968225 32593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:00.968264 32593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:00.968305 32593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:20:00.968329 32593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:20:00.974089 32593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:39867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:00.975299 32593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:00.976389 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:00.985733 32598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:00.985733 32601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:00.985760 32599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:00.986177 32593 server_base.cc:1034] running on GCE node
I20260923 17:20:00.986840 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:00.988094 32593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:00.989339 32593 hybrid_clock.cc:648] HybridClock initialized: now 1790184000989321 us; error 40 us; skew 500 ppm
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184000, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:01.002496 32593 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:20:01.003691 32593 webserver.cc:460] Webserver started at http://127.23.214.190:46395/ using document root <none> and password file <none>
I20260923 17:20:01.004338 32593 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:01.004442 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:01.004693 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:01.007064 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "e655df85c7444ccfa211e35f79fd1645"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "00084444ea64de64211044742a49e4f3"
server_key_iv: "57e7a1ed71d954a53483669663ddbdb3"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.007702 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "e655df85c7444ccfa211e35f79fd1645"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "00084444ea64de64211044742a49e4f3"
server_key_iv: "57e7a1ed71d954a53483669663ddbdb3"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.012063 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:20:01.014874 32608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.016386 32593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:20:01.016572 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "e655df85c7444ccfa211e35f79fd1645"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "00084444ea64de64211044742a49e4f3"
server_key_iv: "57e7a1ed71d954a53483669663ddbdb3"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.016708 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:01.038605 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:01.042374 32593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:01.042598 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:01.050433 32593 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:39867
I20260923 17:20:01.050442 32660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:39867 every 8 connection(s)
I20260923 17:20:01.051481 32593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:20:01.053741 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 32593
I20260923 17:20:01.053879 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:20:01.054185 24410 external_mini_cluster.cc:1428] Setting key 2a226e6ec04ef44e0b3a6e5e0063ced9
I20260923 17:20:01.054523 32661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:20:01.060913 32661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Bootstrap starting.
I20260923 17:20:01.063522 32661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Neither blocks nor log segments found. Creating new log.
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:20:01.064472 32661 log.cc:824] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:01.066603 32661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: No bootstrap required, opened a new log
I20260923 17:20:01.069424 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:01.055650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.23.214.190:39867)
0923 17:20:01.056164 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:01.056181 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:01.056219 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:20:01.057319 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:01.057339 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:01.057682 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:01.058130 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:01.058987 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.060024 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.060894 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.061473 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.061648 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:01.064526 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:01.064531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:01.064546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:01.064601 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:01.066494 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.067035 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.067040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.067044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.067165 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.067464 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.067471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.067472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.067870 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:01.068068 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:01.068388 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 17:20:01.068670 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":153,"threads_started":1}
I20260923 17:20:01.070631 32661 raft_consensus.cc:348] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:01.070915 32661 raft_consensus.cc:374] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:20:01.071008 32661 raft_consensus.cc:729] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e655df85c7444ccfa211e35f79fd1645, State: Initialized, Role: FOLLOWER
I20260923 17:20:01.071511 32661 consensus_queue.cc:260] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:01.071682 32661 raft_consensus.cc:388] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:20:01.071803 32661 raft_consensus.cc:482] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:20:01.071911 32661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:20:01.072919 32661 raft_consensus.cc:504] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:01.073302 32661 leader_election.cc:302] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e655df85c7444ccfa211e35f79fd1645; no voters:
I20260923 17:20:01.073635 32661 leader_election.cc:288] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:20:01.073783 32666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:20:01.074065 32666 raft_consensus.cc:686] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 LEADER]: Becoming Leader. State: Replica: e655df85c7444ccfa211e35f79fd1645, State: Running, Role: LEADER
I20260923 17:20:01.074414 32666 consensus_queue.cc:237] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:01.074958 32661 sys_catalog.cc:564] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:20:01.076270 32667 sys_catalog.cc:455] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e655df85c7444ccfa211e35f79fd1645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } } }
I20260923 17:20:01.076406 32667 sys_catalog.cc:458] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:01.076781 32668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e655df85c7444ccfa211e35f79fd1645. Latest consensus state: current_term: 1 leader_uuid: "e655df85c7444ccfa211e35f79fd1645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } } }
I20260923 17:20:01.076944 32668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:01.077684 32675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:20:01.080453 32675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:20:01.087193 32675 catalog_manager.cc:1347] Generated new cluster ID: 584687b7d9bd4c04ac686d8e163278d7
I20260923 17:20:01.087292 32675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:20:01.113960 32675 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:20:01.115118 32675 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:20:01.121709 32675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Generated new TSK 0
I20260923 17:20:01.122498 32675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:20:01.157560 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:39867
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:01.276724 32687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:01.277015 32687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:01.277061 32687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:01.281260 32687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:01.281334 32687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:01.281396 32687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:20:01.287118 32687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:01.288268 32687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:01.289319 32687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:01.300495 32693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:01.300529 32695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:01.300506 32687 server_base.cc:1034] running on GCE node
W20260923 17:20:01.300509 32692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:01.301209 32687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:01.302057 32687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:01.303238 32687 hybrid_clock.cc:648] HybridClock initialized: now 1790184001303221 us; error 35 us; skew 500 ppm
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:01.306349 32687 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:20:01.307320 32687 webserver.cc:460] Webserver started at http://127.23.214.129:33607/ using document root <none> and password file <none>
I20260923 17:20:01.307865 32687 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:01.307926 32687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:01.308110 32687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:01.310302 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "78db3b5cf028461a8838db0a55845f2b"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "c9eec490e2898b2c527007a171329b79"
server_key_iv: "4fb4f0437681f1dca0a116644c0f19fd"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.310842 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "78db3b5cf028461a8838db0a55845f2b"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "c9eec490e2898b2c527007a171329b79"
server_key_iv: "4fb4f0437681f1dca0a116644c0f19fd"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.314852 32687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:20:01.317490 32702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.318925 32687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:20:01.319108 32687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "78db3b5cf028461a8838db0a55845f2b"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "c9eec490e2898b2c527007a171329b79"
server_key_iv: "4fb4f0437681f1dca0a116644c0f19fd"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.319234 32687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:01.351493 32687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:01.355440 32687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:01.355666 32687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:01.356298 32687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:01.357259 32687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:01.357339 32687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.357416 32687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:01.357468 32687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.366988 32687 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:36029
I20260923 17:20:01.367010 347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:36029 every 8 connection(s)
I20260923 17:20:01.368000 32687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:20:01.375176 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 32687
I20260923 17:20:01.375294 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:20:01.375679 24410 external_mini_cluster.cc:1428] Setting key e3c4eebac8a3a106785a2d8b5b18b153
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:20:01.384373 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:01.370067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:57125 (local address 127.23.214.190:39867)
0923 17:20:01.370219 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:01.370222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:01.371621 (+ 1399us) server_negotiation.cc:407] Connection header received
0923 17:20:01.372755 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:01.372758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:01.372822 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:01.372946 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:01.374229 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.374916 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.376594 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.376942 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.377038 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:01.379832 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:01.379847 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:01.379851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:01.379888 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:01.381519 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.382316 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.382319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.382321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.382409 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.382927 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.382930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.382931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.383143 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:01.383226 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:01.384021 (+ 795us) server_negotiation.cc:299] Negotiation successful
0923 17:20:01.384197 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:20:01.385598 350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:01.370496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:57125)
0923 17:20:01.371441 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:01.371489 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:01.372363 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:01.373130 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:01.373137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:01.373521 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:01.374063 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:01.374078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.375069 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.375072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:01.376467 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:01.376477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.377113 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.377119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:01.377371 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:01.378102 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:01.378120 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:01.379638 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:01.381666 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:01.381674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:01.381696 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:01.382175 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:01.382535 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:01.382541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:01.382546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:01.382774 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:01.383259 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:01.383269 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:01.383678 (+ 409us) client_negotiation.cc:769] Sending connection context
0923 17:20:01.383931 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 17:20:01.384357 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":570,"threads_started":1}
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:20:01.387086 348 heartbeater.cc:347] Connected to a master server at 127.23.214.190:39867
I20260923 17:20:01.387357 348 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:01.387904 348 heartbeater.cc:511] Master 127.23.214.190:39867 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:20:01.389669 32619 ts_manager.cc:194] Registered new tserver with Master: 78db3b5cf028461a8838db0a55845f2b (127.23.214.129:36029)
I20260923 17:20:01.391400 32619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:57125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:20:01.405290 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:39867
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:01.525722 353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:01.526057 353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:01.526106 353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:01.530090 353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:01.530148 353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:01.530205 353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:20:01.535543 353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:01.536607 353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:01.537599 353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:01.547519 361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:01.547536 358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:01.547536 359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:01.547740 353 server_base.cc:1034] running on GCE node
I20260923 17:20:01.548276 353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:01.548961 353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:01.550144 353 hybrid_clock.cc:648] HybridClock initialized: now 1790184001550122 us; error 30 us; skew 500 ppm
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:01.552912 353 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:20:01.553812 353 webserver.cc:460] Webserver started at http://127.23.214.130:42613/ using document root <none> and password file <none>
I20260923 17:20:01.554419 353 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:01.554512 353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:01.554746 353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:01.556792 353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "4be20d96fb494f06aa0b28687ad8bf9a"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "02c11823275c77faf94269aaf60c3834"
server_key_iv: "599061c2a9ccd2f0c189b4edc4492790"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.557358 353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "4be20d96fb494f06aa0b28687ad8bf9a"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "02c11823275c77faf94269aaf60c3834"
server_key_iv: "599061c2a9ccd2f0c189b4edc4492790"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.561144 353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:20:01.563716 368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.565028 353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:20:01.565173 353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "4be20d96fb494f06aa0b28687ad8bf9a"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "02c11823275c77faf94269aaf60c3834"
server_key_iv: "599061c2a9ccd2f0c189b4edc4492790"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.565289 353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:01.584015 353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:01.587108 353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:01.587308 353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:01.587913 353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:01.588752 353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:01.588822 353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.588896 353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:01.588948 353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.598213 353 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:45035
I20260923 17:20:01.598259 481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:45035 every 8 connection(s)
I20260923 17:20:01.599187 353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:20:01.603006 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 353
I20260923 17:20:01.603140 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:20:01.603430 24410 external_mini_cluster.cc:1428] Setting key 28eb32090d765dd0d3684380dc26121e
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:20:01.614996 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:01.601040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:39875 (local address 127.23.214.190:39867)
0923 17:20:01.601190 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:01.601193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:01.602068 (+ 875us) server_negotiation.cc:407] Connection header received
0923 17:20:01.603698 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:01.603701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:01.603750 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:01.603815 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:01.605244 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.605781 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.607315 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.607552 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.607667 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:01.610593 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:01.610601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:01.610606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:01.610642 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:01.612457 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.613062 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.613065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.613067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.613162 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.613641 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.613645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.613646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.613916 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:01.614049 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:01.614643 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 17:20:01.614800 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
I20260923 17:20:01.615880 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:01.601395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:39875)
0923 17:20:01.601859 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:01.601959 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:01.603447 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:01.604033 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:01.604042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:01.604466 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:01.605087 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:01.605108 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.605948 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.605955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:01.607184 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:01.607198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.607702 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.607706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:01.607891 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:01.608526 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:01.608543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:01.610396 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:01.612593 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:01.612599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:01.612613 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:01.612929 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:01.613294 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:01.613297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:01.613298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:01.613501 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:01.614045 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:01.614053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:01.614369 (+ 316us) client_negotiation.cc:769] Sending connection context
0923 17:20:01.614582 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 17:20:01.614941 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260923 17:20:01.617136 482 heartbeater.cc:347] Connected to a master server at 127.23.214.190:39867
I20260923 17:20:01.617399 482 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:01.618072 482 heartbeater.cc:511] Master 127.23.214.190:39867 requested a full tablet report, sending...
I20260923 17:20:01.619279 32619 ts_manager.cc:194] Registered new tserver with Master: 4be20d96fb494f06aa0b28687ad8bf9a (127.23.214.130:45035)
I20260923 17:20:01.620188 32619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:39875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:20:01.634298 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:39867
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:01.747555 487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:01.747839 487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:01.747884 487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:01.752135 487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:01.752202 487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:01.752292 487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:20:01.757720 487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:01.758898 487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:01.759928 487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:01.770232 495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:01.770197 492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:01.770349 487 server_base.cc:1034] running on GCE node
W20260923 17:20:01.770560 493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:01.771031 487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:01.771754 487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:01.772967 487 hybrid_clock.cc:648] HybridClock initialized: now 1790184001772947 us; error 41 us; skew 500 ppm
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:01.776409 487 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:20:01.777477 487 webserver.cc:460] Webserver started at http://127.23.214.131:33511/ using document root <none> and password file <none>
I20260923 17:20:01.778174 487 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:01.778278 487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:01.778535 487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:01.780891 487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "488fa5da3fed40be9f32bbb206647ef2"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "e7cc76722a3c5ff055e8a94f51f1a7ee"
server_key_iv: "f53715e27627b43519fa444bbdfdc833"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.781534 487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "488fa5da3fed40be9f32bbb206647ef2"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "e7cc76722a3c5ff055e8a94f51f1a7ee"
server_key_iv: "f53715e27627b43519fa444bbdfdc833"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.785789 487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 17:20:01.788511 502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.789839 487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:20:01.790079 487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "488fa5da3fed40be9f32bbb206647ef2"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "e7cc76722a3c5ff055e8a94f51f1a7ee"
server_key_iv: "f53715e27627b43519fa444bbdfdc833"
server_key_version: "encryptionkey@0"
I20260923 17:20:01.790212 487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:01.807036 487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:01.810648 487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:01.810899 487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:01.811563 487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:01.812466 487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:01.812517 487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.812592 487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:01.812633 487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:01.822907 487 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:33191
I20260923 17:20:01.822974 615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:33191 every 8 connection(s)
I20260923 17:20:01.824064 487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:20:01.831614 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 487
I20260923 17:20:01.831780 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:20:01.832104 24410 external_mini_cluster.cc:1428] Setting key cde65c58001675da7fc283657bdb8dc4
Sep 23 17:20:01 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:20:01.839128 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:01.826163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:52109 (local address 127.23.214.190:39867)
0923 17:20:01.826316 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:01.826321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:01.827305 (+ 984us) server_negotiation.cc:407] Connection header received
0923 17:20:01.828631 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:01.828634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:01.828683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:01.828765 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:01.830072 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.830604 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.831809 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.832085 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.832203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:01.835293 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:01.835296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:01.835298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:01.835323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:01.836717 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.837277 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.837280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.837281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.837373 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:01.837804 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:01.837806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:01.837807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:01.838161 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:01.838256 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:01.838863 (+ 607us) server_negotiation.cc:299] Negotiation successful
0923 17:20:01.838990 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:20:01.840065 618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:01.826582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:52109)
0923 17:20:01.827076 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:01.827122 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:01.828385 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:01.828934 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:01.828941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:01.829309 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:01.829834 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:01.829848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.830716 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.830719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:01.831661 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:01.831671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:01.832264 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:01.832268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:01.832494 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:01.833233 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:01.833254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:01.835137 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:01.836844 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:01.836850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:01.836864 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:01.837162 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:01.837489 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:01.837492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:01.837494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:01.837658 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:01.838273 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:01.838279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:01.838579 (+ 300us) client_negotiation.cc:769] Sending connection context
0923 17:20:01.838767 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 17:20:01.839085 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260923 17:20:01.841432 616 heartbeater.cc:347] Connected to a master server at 127.23.214.190:39867
I20260923 17:20:01.841734 616 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:01.842371 616 heartbeater.cc:511] Master 127.23.214.190:39867 requested a full tablet report, sending...
I20260923 17:20:01.843526 32619 ts_manager.cc:194] Registered new tserver with Master: 488fa5da3fed40be9f32bbb206647ef2 (127.23.214.131:33191)
I20260923 17:20:01.844125 32619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:52109
I20260923 17:20:01.846740 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab.
Entry for principal oryx/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab.
WARNING: no policy specified for oryx/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab.
Entry for principal oryx/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab.
WARNING: no policy specified for oryx/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab.
Entry for principal oryx/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab.
I20260923 17:20:01.933743 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 32687
I20260923 17:20:01.939585 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 353
I20260923 17:20:01.945120 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 487
I20260923 17:20:01.950879 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 32593
I20260923 17:20:01.957602 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:39867
--webserver_interface=127.23.214.190
--webserver_port=46395
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:39867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:02.069936 625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:02.070245 625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:02.070294 625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:02.074420 625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:20:02.074484 625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:02.074505 625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:02.074522 625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:20:02.074537 625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:20:02.079952 625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:39867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=46395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:02.081252 625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:02.082436 625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:02.091202 633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:02.091192 630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:02.091375 625 server_base.cc:1034] running on GCE node
W20260923 17:20:02.091192 631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:02.092106 625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:02.093225 625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:02.094390 625 hybrid_clock.cc:648] HybridClock initialized: now 1790184002094371 us; error 29 us; skew 500 ppm
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184002, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:02.097210 625 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:20:02.098234 625 webserver.cc:460] Webserver started at http://127.23.214.190:46395/ using document root <none> and password file <none>
I20260923 17:20:02.098821 625 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:02.098896 625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:02.103106 625 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:20:02.105361 640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.106652 625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:20:02.106791 625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "e655df85c7444ccfa211e35f79fd1645"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "00084444ea64de64211044742a49e4f3"
server_key_iv: "57e7a1ed71d954a53483669663ddbdb3"
server_key_version: "encryptionkey@0"
I20260923 17:20:02.107367 625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:02.133973 625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:02.137269 625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:02.137491 625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:02.144796 625 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:39867
I20260923 17:20:02.144821 692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:39867 every 8 connection(s)
I20260923 17:20:02.145862 625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:20:02.150337 693 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:20:02.152194 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Bootstrap starting.
I20260923 17:20:02.154559 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 625
I20260923 17:20:02.155133 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:36029
--local_ip_for_outbound_sockets=127.23.214.129
--tserver_master_addrs=127.23.214.190:39867
--webserver_port=33607
--webserver_interface=127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:20:02.170320 693 log.cc:824] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:02.177070 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:20:02.177518 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Bootstrap complete.
I20260923 17:20:02.181843 693 raft_consensus.cc:348] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:02.182330 693 raft_consensus.cc:729] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e655df85c7444ccfa211e35f79fd1645, State: Initialized, Role: FOLLOWER
I20260923 17:20:02.182821 693 consensus_queue.cc:260] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:02.182991 693 raft_consensus.cc:388] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:20:02.183086 693 raft_consensus.cc:482] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:20:02.183209 693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:20:02.185134 693 raft_consensus.cc:504] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:02.185474 693 leader_election.cc:302] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e655df85c7444ccfa211e35f79fd1645; no voters:
I20260923 17:20:02.185793 693 leader_election.cc:288] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:20:02.185979 697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:20:02.186208 697 raft_consensus.cc:686] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [term 2 LEADER]: Becoming Leader. State: Replica: e655df85c7444ccfa211e35f79fd1645, State: Running, Role: LEADER
I20260923 17:20:02.186515 697 consensus_queue.cc:237] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } }
I20260923 17:20:02.187052 693 sys_catalog.cc:564] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:20:02.188032 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e655df85c7444ccfa211e35f79fd1645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } } }
I20260923 17:20:02.188179 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:02.188421 699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e655df85c7444ccfa211e35f79fd1645. Latest consensus state: current_term: 2 leader_uuid: "e655df85c7444ccfa211e35f79fd1645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e655df85c7444ccfa211e35f79fd1645" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39867 } } }
I20260923 17:20:02.188498 699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645 [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:02.188521 706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:20:02.191747 706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:20:02.192410 706 catalog_manager.cc:1259] Loaded cluster ID: 584687b7d9bd4c04ac686d8e163278d7
I20260923 17:20:02.192468 706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:20:02.194746 706 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:20:02.195403 706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e655df85c7444ccfa211e35f79fd1645: Loaded TSK: 0
I20260923 17:20:02.196625 706 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:20:02.271358 695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:02.271625 695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:02.271672 695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:02.275803 695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:02.275883 695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:02.275944 695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:20:02.281383 695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.129.keytab
--principal=oryx/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:36029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=33607
--tserver_master_addrs=127.23.214.190:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:02.282508 695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:02.283638 695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:02.293984 722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:02.294086 719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:02.294164 720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:02.294513 695 server_base.cc:1034] running on GCE node
I20260923 17:20:02.295028 695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:02.295876 695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:02.297106 695 hybrid_clock.cc:648] HybridClock initialized: now 1790184002297081 us; error 32 us; skew 500 ppm
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184002, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:02.300918 695 init.cc:377] Logged in from keytab as oryx/127.23.214.129@KRBTEST.COM (short username oryx)
I20260923 17:20:02.302263 695 webserver.cc:460] Webserver started at http://127.23.214.129:33607/ using document root <none> and password file <none>
I20260923 17:20:02.303048 695 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:02.303169 695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:02.308475 695 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 17:20:02.311291 729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.312865 695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:20:02.313014 695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "78db3b5cf028461a8838db0a55845f2b"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "c9eec490e2898b2c527007a171329b79"
server_key_iv: "4fb4f0437681f1dca0a116644c0f19fd"
server_key_version: "encryptionkey@0"
I20260923 17:20:02.313755 695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:02.329411 695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:02.333405 695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:02.333673 695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:02.334568 695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:02.335817 695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:02.335897 695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.335951 695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:02.335981 695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.346678 695 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:36029
I20260923 17:20:02.346724 842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:36029 every 8 connection(s)
I20260923 17:20:02.347965 695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:20:02.352315 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 695
I20260923 17:20:02.352823 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:45035
--local_ip_for_outbound_sockets=127.23.214.130
--tserver_master_addrs=127.23.214.190:39867
--webserver_port=42613
--webserver_interface=127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:20:02.362896 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.350285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:41907 (local address 127.23.214.190:39867)
0923 17:20:02.350785 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.350802 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.351116 (+ 314us) server_negotiation.cc:407] Connection header received
0923 17:20:02.352950 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.352969 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.353400 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.353776 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.355281 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.356247 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.357449 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.358097 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.358269 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.362106 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:41907: BlockingRecv error: recv got EOF from 127.23.214.129:41907 (error 108)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":121,"threads_started":1}
I20260923 17:20:02.362921 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.350415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:41907)
0923 17:20:02.350947 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.351000 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.352479 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.353842 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.353849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.354374 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.355089 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.355109 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.356419 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.356422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.357307 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.357316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.358277 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.358281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.358502 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.359161 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.359177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.361846 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":126,"threads_started":1}
W20260923 17:20:02.363406 843 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:39867 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.214.190:39867: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:02.369751 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.363903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:56619)
0923 17:20:02.364025 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.364042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.364115 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.364346 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.364348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.364480 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.364576 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.364581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.365627 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.365629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.366269 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.366278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.366907 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.366913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.366998 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.367521 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.367532 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.369639 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:20:02.369973 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.363968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:56619 (local address 127.23.214.190:39867)
0923 17:20:02.364116 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.364122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.364139 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:20:02.364186 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.364189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.364237 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.364326 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.364682 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.365518 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.366383 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.366741 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.366865 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.369833 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:56619: BlockingRecv error: recv got EOF from 127.23.214.129:56619 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:02.376014 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.370547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:37337)
0923 17:20:02.370670 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.370686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.370768 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.371016 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.371018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.371142 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.371216 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.371221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.372236 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.372239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.372752 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.372759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.373302 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.373304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.373362 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.373986 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.373999 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.375905 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:20:02.376323 843 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:20:02.376336 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.370636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:37337 (local address 127.23.214.190:39867)
0923 17:20:02.370767 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.370774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.370793 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:20:02.370839 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.370845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.370903 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.370999 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.371330 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.372131 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.372856 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.373198 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.373304 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.376070 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:37337: BlockingRecv error: recv got EOF from 127.23.214.129:37337 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260923 17:20:02.472707 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:02.473006 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:02.473052 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:02.477147 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:02.477219 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:02.477283 847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:20:02.482687 847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.130.keytab
--principal=oryx/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:45035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=42613
--tserver_master_addrs=127.23.214.190:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:02.483709 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:02.484803 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:02.494967 852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:02.495477 855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:02.495563 853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:02.495764 847 server_base.cc:1034] running on GCE node
I20260923 17:20:02.496201 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:02.496980 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:02.498184 847 hybrid_clock.cc:648] HybridClock initialized: now 1790184002498144 us; error 54 us; skew 500 ppm
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184002, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:02.501317 847 init.cc:377] Logged in from keytab as oryx/127.23.214.130@KRBTEST.COM (short username oryx)
I20260923 17:20:02.502337 847 webserver.cc:460] Webserver started at http://127.23.214.130:42613/ using document root <none> and password file <none>
I20260923 17:20:02.502939 847 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:02.503015 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:02.506965 847 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:20:02.509042 862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.510448 847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:20:02.510578 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "4be20d96fb494f06aa0b28687ad8bf9a"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "02c11823275c77faf94269aaf60c3834"
server_key_iv: "599061c2a9ccd2f0c189b4edc4492790"
server_key_version: "encryptionkey@0"
I20260923 17:20:02.511214 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:02.542125 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:02.546149 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:02.546375 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:02.547048 847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:02.547916 847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:02.547964 847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.548026 847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:02.548061 847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.557596 847 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:45035
I20260923 17:20:02.557610 975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:45035 every 8 connection(s)
I20260923 17:20:02.558768 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:20:02.560374 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 847
I20260923 17:20:02.560958 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:33191
--local_ip_for_outbound_sockets=127.23.214.131
--tserver_master_addrs=127.23.214.190:39867
--webserver_port=33511
--webserver_interface=127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:20:02.571923 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.560771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:32867 (local address 127.23.214.190:39867)
0923 17:20:02.560940 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.560944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.561995 (+ 1051us) server_negotiation.cc:407] Connection header received
0923 17:20:02.563539 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.563543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.563610 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.563705 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.565133 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.566109 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.567291 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.567731 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.567895 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.571721 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:32867: BlockingRecv error: recv got EOF from 127.23.214.130:32867 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:20:02.572408 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.561241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:32867)
0923 17:20:02.561784 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.561847 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.563291 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.564047 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.564054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.564441 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.564970 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.564985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.566245 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.566247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.567155 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.567163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.567929 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.567931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.568083 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.568800 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.568826 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.571496 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
W20260923 17:20:02.573028 976 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:39867 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.214.190:39867: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:02.579509 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.573518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:35795)
0923 17:20:02.573649 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.573664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.573746 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.574049 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.574052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.574238 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.574365 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.574373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.575128 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.575132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.576054 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.576065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.576541 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.576544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.576641 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.577120 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.577135 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.579369 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:20:02.579962 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.573638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:35795 (local address 127.23.214.190:39867)
0923 17:20:02.573748 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.573752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.573775 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:20:02.573850 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.573852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.573944 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.574012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.574503 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.575007 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.576185 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.576432 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.576513 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.579510 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:35795: BlockingRecv error: recv got EOF from 127.23.214.130:35795 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:02.585378 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.580107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:58095)
0923 17:20:02.580227 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.580242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.580357 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.580680 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.580682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.580839 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.580910 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.580915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.581642 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.581650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.582233 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.582241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.582718 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.582722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.582806 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.583362 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.583372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.585204 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:20:02.585588 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.580350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:58095 (local address 127.23.214.190:39867)
0923 17:20:02.580494 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.580498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.580508 (+ 10us) server_negotiation.cc:407] Connection header received
0923 17:20:02.580539 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.580541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.580577 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.580663 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.581021 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.581524 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.582382 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.582609 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.582718 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.585336 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:58095: BlockingRecv error: recv got EOF from 127.23.214.130:58095 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 17:20:02.585675 976 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:20:02.686844 979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:02.687240 979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:02.687361 979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:02.692893 979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:02.692975 979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:02.693101 979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:20:02.699277 979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/krb5kdc/oryx_127.23.214.131.keytab
--principal=oryx/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:33191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=33511
--tserver_master_addrs=127.23.214.190:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:02.700525 979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:02.701642 979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:02.712013 985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:02.712025 987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:02.711987 984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:02.712414 979 server_base.cc:1034] running on GCE node
I20260923 17:20:02.712857 979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:02.713524 979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:02.714689 979 hybrid_clock.cc:648] HybridClock initialized: now 1790184002714653 us; error 45 us; skew 500 ppm
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184002, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:02.717561 979 init.cc:377] Logged in from keytab as oryx/127.23.214.131@KRBTEST.COM (short username oryx)
I20260923 17:20:02.718647 979 webserver.cc:460] Webserver started at http://127.23.214.131:33511/ using document root <none> and password file <none>
I20260923 17:20:02.719235 979 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:02.719350 979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:02.723369 979 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:20:02.725500 994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.726925 979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:20:02.727025 979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "488fa5da3fed40be9f32bbb206647ef2"
format_stamp: "Formatted at 2026-09-23 17:20:01 on dist-test-slave-9q7f"
server_key: "e7cc76722a3c5ff055e8a94f51f1a7ee"
server_key_iv: "f53715e27627b43519fa444bbdfdc833"
server_key_version: "encryptionkey@0"
I20260923 17:20:02.727566 979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:02.745790 979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:02.749044 979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:02.749212 979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:02.749763 979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:02.750615 979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:02.750666 979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.750705 979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:02.750721 979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:02.760280 979 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:33191
I20260923 17:20:02.760300 1107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:33191 every 8 connection(s)
I20260923 17:20:02.761371 979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:20:02.768837 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 979
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:20:02.773892 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.763409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:45579 (local address 127.23.214.190:39867)
0923 17:20:02.763562 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.763566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.764406 (+ 840us) server_negotiation.cc:407] Connection header received
0923 17:20:02.765819 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.765823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.765906 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.765996 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.767285 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.767844 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.769333 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.769709 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.769846 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.773728 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:45579: BlockingRecv error: recv got EOF from 127.23.214.131:45579 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:20:02.774607 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.763778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:45579)
0923 17:20:02.764232 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.764276 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.765600 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.766204 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.766212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.766591 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.767131 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.767145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.767980 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.767989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.769161 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.769172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.770025 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.770030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.770240 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.770967 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.770985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.773613 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
W20260923 17:20:02.775190 1108 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:39867 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.214.190:39867: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
I20260923 17:20:02.779902 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.770020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.23.214.190:39867)
0923 17:20:02.770289 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.770293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.770306 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:20:02.770353 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.770356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.770403 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.770504 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.771336 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.772029 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.773158 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.773401 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.773508 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.775038 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:02.775043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:02.775057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:02.775084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:02.777160 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:02.777675 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:02.777683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:02.777690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:02.777990 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:02.778336 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:02.778340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:02.778342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:02.778848 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:02.779066 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:02.779394 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 17:20:02.779654 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:02.781531 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.775660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:38929)
0923 17:20:02.775849 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.775865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.775961 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.776248 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.776251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.776414 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.776534 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.776544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.777348 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.777351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.778188 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.778196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.778688 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.778692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.778785 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.779318 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.779331 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.781397 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260923 17:20:02.781592 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.775728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:38929 (local address 127.23.214.190:39867)
0923 17:20:02.775851 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.775854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.775911 (+ 57us) server_negotiation.cc:407] Connection header received
0923 17:20:02.776075 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.776078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.776133 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.776215 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.776684 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.777197 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.778299 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.778571 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.778654 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.781513 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:38929: BlockingRecv error: recv got EOF from 127.23.214.131:38929 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:02 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:02.787317 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:02.782216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:39963)
0923 17:20:02.782382 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:02.782394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:02.782453 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:02.782703 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:02.782706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:02.782864 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:02.782936 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.782941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.783744 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.783750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.784272 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:02.784279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.784737 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.784741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:02.784818 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.785224 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:02.785235 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:02.787161 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260923 17:20:02.787390 845 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:02.782288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:39963 (local address 127.23.214.190:39867)
0923 17:20:02.782416 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:02.782421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:02.782434 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:20:02.782553 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:02.782555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:02.782590 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:02.782686 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:02.783049 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.783638 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.784396 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:02.784628 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:02.784736 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:02.787280 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:39963: BlockingRecv error: recv got EOF from 127.23.214.131:39963 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 17:20:02.787603 1108 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 17:20:03 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:03 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:03.382844 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:03.377052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:60905)
0923 17:20:03.377342 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:03.377356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:03.377457 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:03.377691 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:03.377693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:03.377911 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:03.378010 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:03.378015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.378849 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.378852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:03.379431 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:03.379438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.379963 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.379966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:03.380040 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:03.380653 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:03.380665 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:03.382709 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
W20260923 17:20:03.383224 1114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:03.377082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:60905 (local address 127.23.214.190:39867)
0923 17:20:03.377355 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:03.377358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:03.377409 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:20:03.377524 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:03.377526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:03.377571 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:03.377655 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:03.378126 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.378722 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.379552 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.379837 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.379926 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:03.382909 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:60905: BlockingRecv error: recv got EOF from 127.23.214.129:60905 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
Sep 23 17:20:03 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:03 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:03.592135 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:03.586431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:40243)
0923 17:20:03.586765 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:03.586778 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:03.586891 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:03.587131 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:03.587133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:03.587310 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:03.587410 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:03.587416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.588360 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.588363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:03.589006 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:03.589015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.589451 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.589454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:03.589515 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:03.590147 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:03.590159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:03.592007 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
W20260923 17:20:03.592342 1114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:03.586507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:40243 (local address 127.23.214.190:39867)
0923 17:20:03.586721 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:03.586726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:03.586852 (+ 126us) server_negotiation.cc:407] Connection header received
0923 17:20:03.586977 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:03.586979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:03.587026 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:03.587096 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:03.587576 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.588250 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.589113 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.589351 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.589435 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:03.592202 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:40243: BlockingRecv error: recv got EOF from 127.23.214.130:40243 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 23 17:20:03 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:03 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:03.794765 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:03.788471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:40375)
0923 17:20:03.788871 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:03.788888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:03.789040 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:03.789425 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:03.789428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:03.789664 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:03.789786 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:03.789794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.790770 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.790777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:03.791374 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:03.791382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.791869 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.791872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:03.791952 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:03.792512 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:03.792524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:03.794579 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":95,"threads_started":1}
W20260923 17:20:03.794828 1114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:03.788752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:40375 (local address 127.23.214.190:39867)
0923 17:20:03.788871 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:03.788876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:03.788985 (+ 109us) server_negotiation.cc:407] Connection header received
0923 17:20:03.789164 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:03.789168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:03.789252 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:03.789404 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:03.789960 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.790586 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.791509 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:03.791752 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:03.791865 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:03.794713 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:40375: BlockingRecv error: recv got EOF from 127.23.214.131:40375 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 17:20:04 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:04 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:04.390043 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:04.383751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:52653)
0923 17:20:04.384123 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:04.384137 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:04.384294 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:04.384627 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:04.384629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:04.384817 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:04.384926 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:04.384931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.385687 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.385690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:04.386406 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:04.386417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.386927 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.386930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:04.387004 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:04.387516 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:04.387529 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:04.389796 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":98,"threads_started":1}
W20260923 17:20:04.390151 1118 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:04.383883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:52653 (local address 127.23.214.190:39867)
0923 17:20:04.384133 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:04.384136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:04.384247 (+ 111us) server_negotiation.cc:407] Connection header received
0923 17:20:04.384395 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:04.384398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:04.384444 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:04.384528 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:04.385036 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.385567 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.386525 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.386785 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.386880 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:04.390005 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:52653: BlockingRecv error: recv got EOF from 127.23.214.129:52653 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":63,"threads_started":1}
Sep 23 17:20:04 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:04 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:04.598904 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:04.593013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:52399)
0923 17:20:04.593304 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:04.593323 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:04.593486 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:04.593769 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:04.593772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:04.594020 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:04.594117 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:04.594123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.594951 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.594956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:04.595520 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:04.595527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.596043 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.596047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:04.596132 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:04.596705 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:04.596722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:04.598711 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
W20260923 17:20:04.598981 1118 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:04.593111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:52399 (local address 127.23.214.190:39867)
0923 17:20:04.593272 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:04.593277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:04.593419 (+ 142us) server_negotiation.cc:407] Connection header received
0923 17:20:04.593599 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:04.593601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:04.593658 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:04.593743 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:04.594245 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.594818 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.595638 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.595929 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.596040 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:04.598844 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:52399: BlockingRecv error: recv got EOF from 127.23.214.130:52399 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 17:20:04 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:04 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:04.800872 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:04.795735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:53681)
0923 17:20:04.796006 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:04.796020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:04.796109 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:04.796389 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:04.796392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:04.796566 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:04.796658 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:04.796664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.797392 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.797394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:04.797964 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:04.797972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.798368 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.798370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:04.798427 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:04.798921 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:04.798932 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:04.800751 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
W20260923 17:20:04.801270 1118 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:04.795971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:53681 (local address 127.23.214.190:39867)
0923 17:20:04.796088 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:04.796091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:04.796104 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:20:04.796200 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:04.796202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:04.796249 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:04.796309 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:04.796761 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.797299 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.798062 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:04.798272 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:04.798349 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:04.800898 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:53681: BlockingRecv error: recv got EOF from 127.23.214.131:53681 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 17:20:05 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:05 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:05.396787 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:05.390974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:45739)
0923 17:20:05.391246 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:05.391260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:05.391353 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:05.391654 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:05.391658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:05.391872 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:05.391992 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:05.391999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.392854 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.392859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:05.393533 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:05.393543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.394070 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.394074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:05.394164 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:05.394747 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:05.394759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:05.396609 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
W20260923 17:20:05.396886 1122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:05.391165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:45739 (local address 127.23.214.190:39867)
0923 17:20:05.391409 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:05.391413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:05.391428 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:20:05.391478 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:05.391479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:05.391521 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:05.391599 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:05.392126 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.392743 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.393652 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.393943 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.394067 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:05.396752 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:45739: BlockingRecv error: recv got EOF from 127.23.214.129:45739 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
Sep 23 17:20:05 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:05 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:05.605552 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:05.599721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:43571)
0923 17:20:05.600049 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:05.600066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:05.600212 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:05.600490 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:05.600493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:05.600694 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:05.600787 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:05.600794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.601574 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.601580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:05.602213 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:05.602221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.602681 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.602685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:05.602770 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:05.603335 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:05.603346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:05.605376 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":86,"threads_started":1}
W20260923 17:20:05.605618 1122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:05.599870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:43571 (local address 127.23.214.190:39867)
0923 17:20:05.600039 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:05.600044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:05.600150 (+ 106us) server_negotiation.cc:407] Connection header received
0923 17:20:05.600320 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:05.600323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:05.600378 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:05.600468 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:05.600903 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.601462 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.602349 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.602569 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.602679 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:05.605510 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:43571: BlockingRecv error: recv got EOF from 127.23.214.130:43571 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:20:05 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:05 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:05.807250 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:05.801728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:45075)
0923 17:20:05.802029 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:05.802042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:05.802138 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:05.802401 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:05.802403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:05.802575 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:05.802673 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:05.802678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.803398 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.803400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:05.803950 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:05.803957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.804369 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.804371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:05.804426 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:05.804983 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:05.804995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:05.807037 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
W20260923 17:20:05.807370 1122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:05.801856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:45075 (local address 127.23.214.190:39867)
0923 17:20:05.802024 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:05.802030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:05.802114 (+ 84us) server_negotiation.cc:407] Connection header received
0923 17:20:05.802222 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:05.802224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:05.802270 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:05.802341 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:05.802774 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.803290 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.804055 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:05.804271 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:05.804347 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:05.807193 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:45075: BlockingRecv error: recv got EOF from 127.23.214.131:45075 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 23 17:20:06 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:06 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:06.409350 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:06.397746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:50245)
0923 17:20:06.398258 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:06.398273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:06.398381 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:06.398669 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:06.398672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:06.398885 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:06.398981 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:06.398987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.399761 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.399766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:06.400328 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:06.400335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.400833 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.400837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:06.400926 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:06.407102 (+ 6176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:06.407118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:06.409124 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":123,"threads_started":1}
W20260923 17:20:06.409592 1126 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:06.397808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:50245 (local address 127.23.214.190:39867)
0923 17:20:06.398147 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:06.398151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:06.398358 (+ 207us) server_negotiation.cc:407] Connection header received
0923 17:20:06.398480 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:06.398483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:06.398539 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:06.398645 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:06.399103 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.399646 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.400448 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.400696 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.400823 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:06.409304 (+ 8481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:50245: BlockingRecv error: recv got EOF from 127.23.214.129:50245 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":158,"threads_started":1}
Sep 23 17:20:06 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:06 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:06.612279 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:06.606413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:46717)
0923 17:20:06.606726 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:06.606740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:06.606846 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:06.607073 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:06.607075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:06.607264 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:06.607406 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:06.607411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.608154 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.608156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:06.608735 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:06.608742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.609156 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.609158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:06.609229 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:06.609801 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:06.609814 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:06.612097 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
W20260923 17:20:06.612411 1126 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:06.606496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:46717 (local address 127.23.214.190:39867)
0923 17:20:06.606654 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:06.606661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:06.606815 (+ 154us) server_negotiation.cc:407] Connection header received
0923 17:20:06.606920 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:06.606922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:06.606968 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:06.607036 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:06.607529 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.608061 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.608845 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.609060 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.609138 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:06.612244 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:46717: BlockingRecv error: recv got EOF from 127.23.214.130:46717 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:20:06 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:06 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:06.814318 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:06.808265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:49963)
0923 17:20:06.808560 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:06.808574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:06.808687 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:06.809029 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:06.809032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:06.809279 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:06.809389 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:06.809396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.810268 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.810271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:06.810902 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:06.810912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.811442 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.811444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:06.811507 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:06.812199 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:06.812211 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:06.814199 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
W20260923 17:20:06.814554 1126 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:06.808394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:49963 (local address 127.23.214.190:39867)
0923 17:20:06.808531 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:06.808535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:06.808630 (+ 95us) server_negotiation.cc:407] Connection header received
0923 17:20:06.808828 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:06.808831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:06.808889 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:06.808965 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:06.809533 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.810141 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.811054 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:06.811286 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:06.811409 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:06.814399 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:49963: BlockingRecv error: recv got EOF from 127.23.214.131:49963 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 17:20:07 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:07 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:07.416603 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:07.410376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:47527)
0923 17:20:07.410725 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:07.410739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:07.410854 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:07.411197 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:07.411202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:07.411399 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:07.411493 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:07.411499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.412399 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.412401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:07.412972 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:07.412979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.413456 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.413459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:07.413529 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:07.414218 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:07.414233 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:07.416409 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
W20260923 17:20:07.416711 1130 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:07.410450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:47527 (local address 127.23.214.190:39867)
0923 17:20:07.410754 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:07.410757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:07.410818 (+ 61us) server_negotiation.cc:407] Connection header received
0923 17:20:07.410973 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:07.410977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:07.411035 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:07.411123 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:07.411613 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.412280 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.413075 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.413325 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.413415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:07.416560 (+ 3145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:47527: BlockingRecv error: recv got EOF from 127.23.214.129:47527 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":69,"threads_started":1}
Sep 23 17:20:07 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:07 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:07.619215 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:07.613203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:56123)
0923 17:20:07.613537 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:07.613556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:07.613671 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:07.613982 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:07.613984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:07.614177 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:07.614271 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:07.614277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.615177 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.615180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:07.615808 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:07.615818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.616368 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.616371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:07.616435 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:07.616994 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:07.617006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:07.619091 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
W20260923 17:20:07.619428 1130 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:07.613395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:56123 (local address 127.23.214.190:39867)
0923 17:20:07.613552 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:07.613559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:07.613672 (+ 113us) server_negotiation.cc:407] Connection header received
0923 17:20:07.613743 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:07.613745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:07.613794 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:07.613899 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:07.614442 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.615057 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.615962 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.616256 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.616356 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:07.619291 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:56123: BlockingRecv error: recv got EOF from 127.23.214.130:56123 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 17:20:07 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:07 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:07.821472 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:07.815277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:53307)
0923 17:20:07.815552 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:07.815572 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:07.815690 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:07.815937 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:07.815939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:07.816138 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:07.816233 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:07.816238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.817237 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.817240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:07.818085 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:07.818096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.818579 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.818584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:07.818675 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:07.819235 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:07.819247 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:07.821338 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
W20260923 17:20:07.821650 1130 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:07.815370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:53307 (local address 127.23.214.190:39867)
0923 17:20:07.815512 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:07.815515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:07.815644 (+ 129us) server_negotiation.cc:407] Connection header received
0923 17:20:07.815763 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:07.815765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:07.815812 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:07.815892 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:07.816399 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.817103 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.818187 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:07.818442 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:07.818557 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:07.821516 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:53307: BlockingRecv error: recv got EOF from 127.23.214.131:53307 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 17:20:08 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:08 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:08.423734 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:08.417641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:43445)
0923 17:20:08.418137 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:08.418151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:08.418255 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:08.418623 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:08.418629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:08.418845 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:08.418996 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:08.419006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.419953 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.419955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:08.420609 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:08.420618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.421035 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.421037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:08.421100 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:08.421619 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:08.421630 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:08.423615 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":86,"threads_started":1}
W20260923 17:20:08.423906 1134 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:08.417821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:43445 (local address 127.23.214.190:39867)
0923 17:20:08.418137 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:08.418142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:08.418223 (+ 81us) server_negotiation.cc:407] Connection header received
0923 17:20:08.418383 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:08.418386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:08.418473 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:08.418567 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:08.419151 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.419848 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.420714 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.420938 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.421018 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:08.423763 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:43445: BlockingRecv error: recv got EOF from 127.23.214.129:43445 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
Sep 23 17:20:08 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:08 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:08.626073 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:08.620150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:45841)
0923 17:20:08.620567 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:08.620585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:08.620700 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:08.620989 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:08.620991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:08.621173 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:08.621265 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:08.621269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.622128 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.622130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:08.622678 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:08.622685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.623185 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.623189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:08.623290 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:08.624020 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:08.624037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:08.625951 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":71,"threads_started":1}
W20260923 17:20:08.626209 1134 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:08.620352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:45841 (local address 127.23.214.190:39867)
0923 17:20:08.620515 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:08.620519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:08.620690 (+ 171us) server_negotiation.cc:407] Connection header received
0923 17:20:08.620804 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:08.620806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:08.620864 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:08.620956 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:08.621390 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.622030 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.622794 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.623045 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.623133 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:08.626063 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:45841: BlockingRecv error: recv got EOF from 127.23.214.130:45841 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:20:08 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:08 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:08.828536 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:08.822447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:59043)
0923 17:20:08.822863 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:08.822877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:08.823030 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:08.823427 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:08.823430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:08.823703 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:08.823837 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:08.823844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.824804 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.824807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:08.825415 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:08.825423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.826006 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.826016 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:08.826086 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:08.826607 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:08.826619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:08.828415 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":86,"threads_started":1}
W20260923 17:20:08.828711 1134 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:08.822596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:59043 (local address 127.23.214.190:39867)
0923 17:20:08.822784 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:08.822789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:08.822991 (+ 202us) server_negotiation.cc:407] Connection header received
0923 17:20:08.823129 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:08.823132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:08.823211 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:08.823363 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:08.823961 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.824693 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.825558 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:08.825805 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:08.825937 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:08.828571 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:59043: BlockingRecv error: recv got EOF from 127.23.214.131:59043 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 17:20:09 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:09 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:09.430578 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:09.424571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:55935)
0923 17:20:09.424914 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:09.424941 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:09.425045 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:09.425369 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:09.425372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:09.425554 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:09.425651 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:09.425657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.426564 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.426567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:09.427160 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:09.427169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.427692 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.427695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:09.427776 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:09.428361 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:09.428374 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:09.430380 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
W20260923 17:20:09.430647 1138 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:09.424753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:55935 (local address 127.23.214.190:39867)
0923 17:20:09.425004 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:09.425008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:09.425022 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:20:09.425144 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:09.425146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:09.425187 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:09.425265 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:09.425763 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.426419 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.427284 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.427560 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.427645 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:09.430518 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:55935: BlockingRecv error: recv got EOF from 127.23.214.129:55935 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
Sep 23 17:20:09 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:09 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:09.632977 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:09.626887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:59499)
0923 17:20:09.627231 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:09.627245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:09.627342 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:09.627586 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:09.627588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:09.627767 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:09.627856 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:09.627862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.628655 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.628658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:09.629278 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:09.629289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.629807 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.629810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:09.629959 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:09.630637 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:09.630651 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:09.632766 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
W20260923 17:20:09.633064 1138 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:09.626972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:59499 (local address 127.23.214.190:39867)
0923 17:20:09.627160 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:09.627167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:09.627327 (+ 160us) server_negotiation.cc:407] Connection header received
0923 17:20:09.627426 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:09.627428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:09.627475 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:09.627547 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:09.627967 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.628521 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.629434 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.629685 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.629773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:09.632938 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:59499: BlockingRecv error: recv got EOF from 127.23.214.130:59499 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 23 17:20:09 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:09 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:09.835016 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:09.829325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:35309)
0923 17:20:09.829643 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:09.829657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:09.829751 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:09.830034 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:09.830036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:09.830212 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:09.830304 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:09.830309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.831089 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.831093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:09.831733 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:09.831750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.832270 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.832274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:09.832360 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:09.832931 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:09.832942 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:09.834882 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
W20260923 17:20:09.835278 1138 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:09.829457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:35309 (local address 127.23.214.190:39867)
0923 17:20:09.829591 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:09.829595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:09.829724 (+ 129us) server_negotiation.cc:407] Connection header received
0923 17:20:09.829830 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:09.829832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:09.829901 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:09.829990 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:09.830414 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.830959 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.831884 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:09.832143 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:09.832251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:09.835122 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:35309: BlockingRecv error: recv got EOF from 127.23.214.131:35309 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 17:20:10 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:10 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:10.437776 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:10.431822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:46237)
0923 17:20:10.432111 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:10.432126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:10.432218 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:10.432526 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:10.432528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:10.432704 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:10.432809 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:10.432814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.433606 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.433609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:10.434221 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:10.434230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.434800 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.434803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:10.434876 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:10.435520 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:10.435539 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:10.437651 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":109,"threads_started":1}
W20260923 17:20:10.438025 1142 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:10.431918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:46237 (local address 127.23.214.190:39867)
0923 17:20:10.432158 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:10.432162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:10.432176 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:20:10.432341 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:10.432344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:10.432391 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:10.432480 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:10.432922 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.433488 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.434385 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.434639 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.434805 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:10.437834 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:46237: BlockingRecv error: recv got EOF from 127.23.214.129:46237 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
Sep 23 17:20:10 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:10 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:10.640091 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:10.634034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:41351)
0923 17:20:10.634356 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:10.634377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:10.634507 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:10.634826 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:10.634830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:10.635019 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:10.635124 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:10.635129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.636168 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.636171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:10.636778 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:10.636787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.637270 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.637273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:10.637346 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:10.638050 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:10.638062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:10.639965 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":123,"threads_started":1}
W20260923 17:20:10.640532 1142 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:10.634196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:41351 (local address 127.23.214.190:39867)
0923 17:20:10.634387 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:10.634392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:10.634476 (+ 84us) server_negotiation.cc:407] Connection header received
0923 17:20:10.634602 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:10.634604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:10.634654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:10.634750 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:10.635261 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.636040 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.636904 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.637154 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.637243 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:10.640190 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:41351: BlockingRecv error: recv got EOF from 127.23.214.130:41351 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 17:20:10 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:10 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:10.842355 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:10.835990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:39215)
0923 17:20:10.836488 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:10.836505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:10.836630 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:10.836964 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:10.836967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:10.837181 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:10.837323 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:10.837330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.838218 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.838221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:10.838786 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:10.838795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.839267 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.839271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:10.839360 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:10.839952 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:10.839965 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:10.842217 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":114,"threads_started":1}
W20260923 17:20:10.842608 1142 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:10.836108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:39215 (local address 127.23.214.190:39867)
0923 17:20:10.836254 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:10.836261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:10.836648 (+ 387us) server_negotiation.cc:407] Connection header received
0923 17:20:10.836745 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:10.836748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:10.836826 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:10.836937 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:10.837457 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.838094 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.838915 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:10.839142 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:10.839254 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:10.842440 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:39215: BlockingRecv error: recv got EOF from 127.23.214.131:39215 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 17:20:11 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:11 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:11.450114 1145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:11.438785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:60255)
0923 17:20:11.439078 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:11.439093 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:11.439192 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:11.439475 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:11.439477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:11.439651 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:11.439746 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:11.439751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.440609 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.440613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:11.441422 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:11.441433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.441974 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.441977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:11.442044 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:11.446867 (+ 4823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:11.446892 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:11.449857 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
W20260923 17:20:11.450343 1146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:11.438906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:60255 (local address 127.23.214.190:39867)
0923 17:20:11.439187 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:11.439191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:11.439205 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:20:11.439289 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:11.439292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:11.439344 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:11.439432 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:11.439866 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.440493 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.441566 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.441844 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.441967 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:11.450158 (+ 8191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:60255: BlockingRecv error: recv got EOF from 127.23.214.129:60255 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
Sep 23 17:20:11 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:11 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:11.647257 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:11.641039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:34099)
0923 17:20:11.641557 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:11.641579 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:11.641728 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:11.642052 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:11.642055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:11.642275 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:11.642371 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:11.642378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.643190 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.643195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:11.643779 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:11.643787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.644287 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.644290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:11.644384 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:11.644918 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:11.644930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:11.647087 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":147,"threads_started":1}
W20260923 17:20:11.647437 1146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:11.641215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:34099 (local address 127.23.214.190:39867)
0923 17:20:11.641524 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:11.641530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:11.641668 (+ 138us) server_negotiation.cc:407] Connection header received
0923 17:20:11.641846 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:11.641849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:11.641925 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:11.642025 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:11.642506 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.643066 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.643919 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.644153 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.644269 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:11.647270 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:34099: BlockingRecv error: recv got EOF from 127.23.214.130:34099 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
Sep 23 17:20:11 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:11 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:11.849114 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:11.843166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:56195)
0923 17:20:11.843526 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:11.843542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:11.843683 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:11.843974 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:11.843977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:11.844188 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:11.844277 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:11.844283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.845051 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.845056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:11.845637 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:11.845644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.846177 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.846181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:11.846262 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:11.846830 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:11.846844 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:11.848922 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
W20260923 17:20:11.849167 1146 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:11.843314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:56195 (local address 127.23.214.190:39867)
0923 17:20:11.843502 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:11.843507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:11.843626 (+ 119us) server_negotiation.cc:407] Connection header received
0923 17:20:11.843802 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:11.843806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:11.843862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:11.843953 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:11.844414 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.844944 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.845761 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:11.846065 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:11.846177 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:11.849057 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:56195: BlockingRecv error: recv got EOF from 127.23.214.131:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 17:20:12 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:12 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:12.457962 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:12.451447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:43061)
0923 17:20:12.451773 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:12.451796 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:12.451921 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:12.452233 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:12.452236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:12.452490 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:12.452609 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:12.452615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.453452 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.453458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:12.454092 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:12.454102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.454601 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.454605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:12.454702 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:12.455335 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:12.455355 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:12.457752 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
W20260923 17:20:12.458045 1150 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:12.451603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:43061 (local address 127.23.214.190:39867)
0923 17:20:12.451898 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:12.451904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:12.451920 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:20:12.452031 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:12.452035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:12.452095 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:12.452211 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:12.452763 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.453323 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.454225 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.454474 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.454600 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:12.457909 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:43061: BlockingRecv error: recv got EOF from 127.23.214.129:43061 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
Sep 23 17:20:12 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:12 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:12.654075 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:12.648081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:53037)
0923 17:20:12.648408 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:12.648422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:12.648517 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:12.648760 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:12.648763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:12.648961 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:12.649058 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:12.649064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.649827 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.649830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:12.650684 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:12.650696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.651142 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.651146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:12.651250 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:12.651826 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:12.651838 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:12.653900 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
W20260923 17:20:12.654177 1150 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:12.648190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:53037 (local address 127.23.214.190:39867)
0923 17:20:12.648334 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:12.648338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:12.648490 (+ 152us) server_negotiation.cc:407] Connection header received
0923 17:20:12.648597 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:12.648599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:12.648643 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:12.648713 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:12.649158 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.649687 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.650797 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.651024 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.651297 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:12.654056 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:53037: BlockingRecv error: recv got EOF from 127.23.214.130:53037 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:20:12 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:12 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:12.856141 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:12.850212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:49497)
0923 17:20:12.850536 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:12.850574 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:12.850698 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:12.851053 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:12.851057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:12.851317 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:12.851418 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:12.851447 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.852394 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.852397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:12.852962 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:12.852969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.853455 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.853458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:12.853530 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:12.854139 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:12.854151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:12.856015 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
W20260923 17:20:12.856271 1150 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:12.850421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:49497 (local address 127.23.214.190:39867)
0923 17:20:12.850583 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:12.850587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:12.850644 (+ 57us) server_negotiation.cc:407] Connection header received
0923 17:20:12.850842 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:12.850845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:12.850906 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:12.851000 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:12.851606 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.852289 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.853086 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:12.853307 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:12.853405 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:12.856151 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:49497: BlockingRecv error: recv got EOF from 127.23.214.131:49497 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 17:20:13 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:13 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:13.465376 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:13.459071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:44707)
0923 17:20:13.459508 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:13.459522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:13.459661 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:13.460110 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:13.460115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:13.460431 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:13.460565 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:13.460576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.461508 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.461510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:13.462156 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:13.462165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.462623 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.462626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:13.462691 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:13.463178 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:13.463189 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:13.465249 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":122,"threads_started":1}
W20260923 17:20:13.465607 1154 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:13.459259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:44707 (local address 127.23.214.190:39867)
0923 17:20:13.459574 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:13.459579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:13.459606 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:20:13.459897 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:13.459900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:13.459956 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:13.460075 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:13.460700 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.461405 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.462258 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.462521 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.462608 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:13.465450 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:44707: BlockingRecv error: recv got EOF from 127.23.214.129:44707 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
Sep 23 17:20:13 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:13 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:13.661350 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:13.654985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:43075)
0923 17:20:13.655327 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:13.655358 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:13.655532 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:13.655841 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:13.655844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:13.656068 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:13.656168 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:13.656174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.657052 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.657057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:13.657700 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:13.657709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.658295 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.658299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:13.658388 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:13.658969 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:13.658982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:13.661147 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
W20260923 17:20:13.661458 1154 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:13.655074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:43075 (local address 127.23.214.190:39867)
0923 17:20:13.655342 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:13.655347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:13.655462 (+ 115us) server_negotiation.cc:407] Connection header received
0923 17:20:13.655644 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:13.655649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:13.655700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:13.655810 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:13.656322 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.656926 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.657858 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.658162 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.658290 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:13.661305 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:43075: BlockingRecv error: recv got EOF from 127.23.214.130:43075 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 23 17:20:13 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:13 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:13.862720 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:13.857158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:46885)
0923 17:20:13.857520 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:13.857542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:13.857639 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:13.857942 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:13.857944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:13.858137 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:13.858235 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:13.858240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.858992 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.858995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:13.859573 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:13.859581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.860047 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.860049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:13.860110 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:13.860633 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:13.860644 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:13.862598 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":170,"threads_started":1}
W20260923 17:20:13.862949 1154 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:13.857155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:46885 (local address 127.23.214.190:39867)
0923 17:20:13.857305 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:13.857311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:13.857624 (+ 313us) server_negotiation.cc:407] Connection header received
0923 17:20:13.857723 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:13.857725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:13.857787 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:13.857919 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:13.858346 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.858886 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.859697 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:13.859943 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:13.860031 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:13.862830 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:46885: BlockingRecv error: recv got EOF from 127.23.214.131:46885 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 17:20:14 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:14 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:14.472625 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:14.466538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:35367)
0923 17:20:14.466895 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:14.466941 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:14.467066 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:14.467430 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:14.467433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:14.467686 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:14.467805 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:14.467811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.468688 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.468690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:14.469254 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:14.469262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.469791 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.469794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:14.469907 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:14.470458 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:14.470470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:14.472481 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":83,"threads_started":1}
W20260923 17:20:14.472909 1158 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:14.466621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:35367 (local address 127.23.214.190:39867)
0923 17:20:14.466897 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:14.466902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:14.467045 (+ 143us) server_negotiation.cc:407] Connection header received
0923 17:20:14.467176 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:14.467179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:14.467245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:14.467374 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:14.467963 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.468585 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.469373 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.469635 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.469731 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:14.472639 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:35367: BlockingRecv error: recv got EOF from 127.23.214.129:35367 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
Sep 23 17:20:14 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:14 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:14.667835 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:14.662377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:33307)
0923 17:20:14.662646 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:14.662659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:14.662763 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:14.662996 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:14.662998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:14.663177 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:14.663270 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:14.663275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.664082 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.664085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:14.664645 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:14.664654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.665089 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.665091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:14.665158 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:14.665743 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:14.665755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:14.667704 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20260923 17:20:14.668234 1158 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:14.662408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:33307 (local address 127.23.214.190:39867)
0923 17:20:14.662563 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:14.662568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:14.662733 (+ 165us) server_negotiation.cc:407] Connection header received
0923 17:20:14.662844 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:14.662846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:14.662892 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:14.662960 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:14.663380 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.663971 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.664756 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.664983 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.665065 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:14.668097 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:33307: BlockingRecv error: recv got EOF from 127.23.214.130:33307 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:20:14 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:14 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:14.869058 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:14.863564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:38791)
0923 17:20:14.863830 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:14.863849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:14.863947 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:14.864308 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:14.864311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:14.864568 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:14.864727 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:14.864736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.865655 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.865658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:14.866239 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:14.866247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.866692 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.866695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:14.866760 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:14.867247 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:14.867258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:14.868945 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
W20260923 17:20:14.869232 1158 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:14.863635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:38791 (local address 127.23.214.190:39867)
0923 17:20:14.863819 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:14.863824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:14.863930 (+ 106us) server_negotiation.cc:407] Connection header received
0923 17:20:14.864066 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:14.864068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:14.864138 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:14.864231 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:14.864885 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.865548 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.866371 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:14.866584 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:14.866663 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:14.869075 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:38791: BlockingRecv error: recv got EOF from 127.23.214.131:38791 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 17:20:15 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:15 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:15.479575 1161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:15.473520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:44709)
0923 17:20:15.473838 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:15.473853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:15.474026 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:15.474282 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:15.474285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:15.474464 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:15.474562 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:15.474567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.475315 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.475318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:15.476161 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:15.476174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.476669 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.476673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:15.476764 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:15.477428 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:15.477442 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:15.479370 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
W20260923 17:20:15.479635 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:15.473652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:44709 (local address 127.23.214.190:39867)
0923 17:20:15.473954 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:15.473957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:15.473973 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:20:15.474119 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:15.474122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:15.474167 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:15.474250 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:15.474662 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.475199 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.476289 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.476547 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.476663 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:15.479503 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:44709: BlockingRecv error: recv got EOF from 127.23.214.129:44709 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":59,"threads_started":1}
Sep 23 17:20:15 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:15 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:15.674561 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:15.668622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:42079)
0923 17:20:15.668949 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:15.668967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:15.669113 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:15.669421 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:15.669424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:15.669639 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:15.669733 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:15.669740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.670598 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.670602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:15.671195 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:15.671204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.671690 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.671694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:15.671780 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:15.672386 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:15.672398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:15.674367 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
W20260923 17:20:15.674618 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:15.668786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:42079 (local address 127.23.214.190:39867)
0923 17:20:15.668949 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:15.668955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:15.669050 (+ 95us) server_negotiation.cc:407] Connection header received
0923 17:20:15.669213 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:15.669217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:15.669267 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:15.669363 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:15.669866 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.670472 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.671327 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.671575 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.671688 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:15.674505 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:42079: BlockingRecv error: recv got EOF from 127.23.214.130:42079 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:20:15 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:15 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:15.875469 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:15.869943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:47093)
0923 17:20:15.870242 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:15.870255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:15.870405 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:15.870636 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:15.870638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:15.870822 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:15.870918 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:15.870924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.871752 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.871755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:15.872307 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:15.872336 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.872788 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.872792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:15.872857 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:15.873431 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:15.873443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:15.875350 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
W20260923 17:20:15.875655 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:15.869993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:47093 (local address 127.23.214.190:39867)
0923 17:20:15.870142 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:15.870146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:15.870304 (+ 158us) server_negotiation.cc:407] Connection header received
0923 17:20:15.870481 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:15.870483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:15.870527 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:15.870594 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:15.871062 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.871643 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.872449 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:15.872663 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:15.872745 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:15.875487 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:47093: BlockingRecv error: recv got EOF from 127.23.214.131:47093 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 17:20:16 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:16 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:16.490670 1165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:16.480559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:57401)
0923 17:20:16.480858 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:16.480878 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:16.480987 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:16.481284 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:16.481288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:16.481512 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:16.481608 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:16.481614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.482554 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.482560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:16.483214 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:16.483224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.483679 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.483681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:16.483749 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:16.488333 (+ 4584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:16.488352 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:16.490474 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
W20260923 17:20:16.490763 1166 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:16.480633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:57401 (local address 127.23.214.190:39867)
0923 17:20:16.480981 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:16.480987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:16.481008 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:20:16.481098 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:16.481100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:16.481145 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:16.481265 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:16.481756 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.482440 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.483320 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.483543 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.483654 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:16.490625 (+ 6971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:57401: BlockingRecv error: recv got EOF from 127.23.214.129:57401 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
Sep 23 17:20:16 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:16 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:16.681022 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:16.675503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:45377)
0923 17:20:16.675798 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:16.675812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:16.675962 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:16.676225 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:16.676227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:16.676420 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:16.676516 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:16.676521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.677299 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.677302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:16.677999 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:16.678009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.678467 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.678470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:16.678532 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:16.679062 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:16.679074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:16.680891 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
W20260923 17:20:16.681339 1166 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:16.675619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:45377 (local address 127.23.214.190:39867)
0923 17:20:16.675775 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:16.675780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:16.675924 (+ 144us) server_negotiation.cc:407] Connection header received
0923 17:20:16.676075 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:16.676077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:16.676124 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:16.676190 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:16.676650 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.677201 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.678123 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.678360 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.678450 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:16.681060 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:45377: BlockingRecv error: recv got EOF from 127.23.214.130:45377 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 17:20:16 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:16 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.131@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:16.882766 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:16.876474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.131:52749)
0923 17:20:16.876786 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:16.876800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:16.876910 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:16.877157 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:16.877159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:16.877387 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:16.877487 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:16.877493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.878617 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.878624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:16.879501 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:16.879515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.880022 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.880026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:16.880100 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:16.880639 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:16.880651 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:16.882632 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":122,"threads_started":1}
W20260923 17:20:16.882943 1166 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:16.876618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:52749 (local address 127.23.214.190:39867)
0923 17:20:16.876775 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:16.876779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:16.876860 (+ 81us) server_negotiation.cc:407] Connection header received
0923 17:20:16.877003 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:16.877006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:16.877053 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:16.877119 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:16.877693 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.878448 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.879640 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:16.879876 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:16.879965 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:16.882805 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.131:52749: BlockingRecv error: recv got EOF from 127.23.214.131:52749 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 17:20:17 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:17 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.129@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:17.497954 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:17.491804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.129:36215)
0923 17:20:17.492109 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:17.492128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:17.492274 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:17.492602 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:17.492607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:17.492847 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:17.492943 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:17.492949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.493984 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.493987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:17.494591 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:17.494600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.495119 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.495122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:17.495202 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:17.495714 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:17.495725 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:17.497744 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
W20260923 17:20:17.498147 1170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:17.491982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:36215 (local address 127.23.214.190:39867)
0923 17:20:17.492269 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:17.492273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:17.492286 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:20:17.492384 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:17.492386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:17.492429 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:17.492550 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:17.493048 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.493772 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.494737 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.494972 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.495056 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:17.497959 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.129:36215: BlockingRecv error: recv got EOF from 127.23.214.129:36215 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
Sep 23 17:20:17 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:20:17 dist-test-slave-9q7f krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.214.130@KRBTEST.COM for oryx/127.23.214.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:20:17.688459 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:17.682119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:39867 (local address 127.23.214.130:59903)
0923 17:20:17.682454 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:17.682468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:17.682565 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:17.682822 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:17.682824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:17.683008 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:17.683113 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:17.683119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.683981 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.683984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:17.684587 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:17.684596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.685099 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.685103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:17.685170 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:17.685755 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:17.685767 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:17.688325 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.214.190:39867: Server oryx/127.23.214.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":81,"threads_started":1}
W20260923 17:20:17.688617 1170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:17.682204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:59903 (local address 127.23.214.190:39867)
0923 17:20:17.682382 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:17.682386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:17.682578 (+ 192us) server_negotiation.cc:407] Connection header received
0923 17:20:17.682632 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:17.682635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:17.682691 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:17.682795 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:17.683231 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.683794 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.684715 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:17.684951 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:17.685036 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:17.688483 (+ 3447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.214.130:59903: BlockingRecv error: recv got EOF from 127.23.214.130:59903 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:20:17.770224 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 695
I20260923 17:20:17.775995 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 847
I20260923 17:20:17.781525 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 979
I20260923 17:20:17.787146 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 625
2026-09-23T17:20:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18126 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T17:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:20:17Z Disabled control of system clock
I20260923 17:20:17.818708 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44633
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:20:17.932025 1175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:17.932329 1175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:17.932365 1175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:17.936672 1175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:20:17.936746 1175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:17.936761 1175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:17.936779 1175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:20:17.936794 1175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:20:17.941807 1175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:44633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:17.943050 1175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:17.944584 1175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:17.953821 1183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:17.954064 1181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:17.953855 1175 server_base.cc:1034] running on GCE node
W20260923 17:20:17.953821 1180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:17.954769 1175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:17.955925 1175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:17.957122 1175 hybrid_clock.cc:648] HybridClock initialized: now 1790184017957097 us; error 37 us; skew 500 ppm
I20260923 17:20:17.959132 1175 webserver.cc:460] Webserver started at http://127.23.214.190:40029/ using document root <none> and password file <none>
I20260923 17:20:17.959743 1175 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:17.959839 1175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:17.960091 1175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:17.962277 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "8a41bed4c37042f88e432d9bc47e82ac"
format_stamp: "Formatted at 2026-09-23 17:20:17 on dist-test-slave-9q7f"
server_key: "cc6fcef4335641083471cafe6673fbb8"
server_key_iv: "4bc018d56197a675c30d8aa1361f25aa"
server_key_version: "encryptionkey@0"
I20260923 17:20:17.962854 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "8a41bed4c37042f88e432d9bc47e82ac"
format_stamp: "Formatted at 2026-09-23 17:20:17 on dist-test-slave-9q7f"
server_key: "cc6fcef4335641083471cafe6673fbb8"
server_key_iv: "4bc018d56197a675c30d8aa1361f25aa"
server_key_version: "encryptionkey@0"
I20260923 17:20:17.966804 1175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:20:17.969511 1189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:17.970856 1175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:20:17.971024 1175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "8a41bed4c37042f88e432d9bc47e82ac"
format_stamp: "Formatted at 2026-09-23 17:20:17 on dist-test-slave-9q7f"
server_key: "cc6fcef4335641083471cafe6673fbb8"
server_key_iv: "4bc018d56197a675c30d8aa1361f25aa"
server_key_version: "encryptionkey@0"
I20260923 17:20:17.971150 1175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:18.003623 1175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:18.004432 1175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:18.004650 1175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:18.012205 1175 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:44633
I20260923 17:20:18.012233 1241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:44633 every 8 connection(s)
I20260923 17:20:18.013196 1175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:20:18.015224 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 1175
I20260923 17:20:18.015362 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:20:18.015667 24410 external_mini_cluster.cc:1428] Setting key e645e4de197c6b221e5be0d44c59d192
I20260923 17:20:18.016899 1242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:20:18.022680 1242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac: Bootstrap starting.
I20260923 17:20:18.025509 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.017092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.23.214.190:44633)
0923 17:20:18.017752 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:18.017775 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:18.017836 (+ 61us) server_negotiation.cc:407] Connection header received
0923 17:20:18.018558 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:18.018575 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:18.018969 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:18.019312 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:18.019734 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.020708 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.021627 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.022192 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.022371 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:18.023329 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:18.023337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:18.023355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:18.023856 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:18.023933 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:18.023966 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:18.024084 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:18.024221 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 17:20:18.024515 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260923 17:20:18.025585 1242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac: Neither blocks nor log segments found. Creating new log.
I20260923 17:20:18.026399 1242 log.cc:824] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:18.028391 1242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac: No bootstrap required, opened a new log
I20260923 17:20:18.032078 1242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a41bed4c37042f88e432d9bc47e82ac" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44633 } }
I20260923 17:20:18.032337 1242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:20:18.032423 1242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a41bed4c37042f88e432d9bc47e82ac, State: Initialized, Role: FOLLOWER
I20260923 17:20:18.032944 1242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a41bed4c37042f88e432d9bc47e82ac" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44633 } }
I20260923 17:20:18.033129 1242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:20:18.033201 1242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:20:18.033303 1242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:20:18.034317 1242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a41bed4c37042f88e432d9bc47e82ac" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44633 } }
I20260923 17:20:18.034678 1242 leader_election.cc:302] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a41bed4c37042f88e432d9bc47e82ac; no voters:
I20260923 17:20:18.034994 1242 leader_election.cc:288] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:20:18.035190 1247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:20:18.035447 1247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [term 1 LEADER]: Becoming Leader. State: Replica: 8a41bed4c37042f88e432d9bc47e82ac, State: Running, Role: LEADER
I20260923 17:20:18.035786 1247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a41bed4c37042f88e432d9bc47e82ac" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44633 } }
I20260923 17:20:18.036357 1242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:20:18.037657 1248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a41bed4c37042f88e432d9bc47e82ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a41bed4c37042f88e432d9bc47e82ac" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44633 } } }
I20260923 17:20:18.037684 1249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a41bed4c37042f88e432d9bc47e82ac. Latest consensus state: current_term: 1 leader_uuid: "8a41bed4c37042f88e432d9bc47e82ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a41bed4c37042f88e432d9bc47e82ac" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44633 } } }
I20260923 17:20:18.037911 1248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:18.037982 1249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:18.038452 1256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:20:18.041811 1256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:20:18.047856 1256 catalog_manager.cc:1347] Generated new cluster ID: b97517f403d9498eb7191e4f14452793
I20260923 17:20:18.047943 1256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:20:18.065164 1256 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:20:18.066207 1256 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:20:18.075275 1256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a41bed4c37042f88e432d9bc47e82ac: Generated new TSK 0
I20260923 17:20:18.076056 1256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:20:18.085655 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44633
--builtin_ntp_servers=127.23.214.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 17:20:18.201498 1266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:18.201809 1266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:18.201916 1266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:18.206043 1266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:18.206137 1266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:18.206233 1266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:20:18.211356 1266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:18.212545 1266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:18.214017 1266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:18.223803 1274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:18.223814 1272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:18.223821 1271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:18.223937 1266 server_base.cc:1034] running on GCE node
I20260923 17:20:18.224576 1266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:18.225257 1266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:18.226442 1266 hybrid_clock.cc:648] HybridClock initialized: now 1790184018226422 us; error 31 us; skew 500 ppm
I20260923 17:20:18.228518 1266 webserver.cc:460] Webserver started at http://127.23.214.129:37303/ using document root <none> and password file <none>
I20260923 17:20:18.229169 1266 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:18.229244 1266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:18.229435 1266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:18.231853 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "8c9d1531900249b58c17c49fedbb1309"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "b0750e8821ba6480b283207dc60320d2"
server_key_iv: "f811a342c2cd85ad9662d74e8a977611"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.232403 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "8c9d1531900249b58c17c49fedbb1309"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "b0750e8821ba6480b283207dc60320d2"
server_key_iv: "f811a342c2cd85ad9662d74e8a977611"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.236476 1266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 17:20:18.239357 1280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.240697 1266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:20:18.240856 1266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "8c9d1531900249b58c17c49fedbb1309"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "b0750e8821ba6480b283207dc60320d2"
server_key_iv: "f811a342c2cd85ad9662d74e8a977611"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.240978 1266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:18.264524 1266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:18.265328 1266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:18.265539 1266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:18.266234 1266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:18.267275 1266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:18.267349 1266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.267417 1266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:18.267460 1266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.277266 1266 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:41347
I20260923 17:20:18.277331 1393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:41347 every 8 connection(s)
I20260923 17:20:18.278281 1266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:20:18.282644 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 1266
I20260923 17:20:18.282791 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:20:18.283098 24410 external_mini_cluster.cc:1428] Setting key 9a5f24a20b904eaa98a90a57ec290af8
I20260923 17:20:18.285104 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44633
--builtin_ntp_servers=127.23.214.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:20:18.288625 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.280196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:54949 (local address 127.23.214.190:44633)
0923 17:20:18.280338 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:18.280343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:18.281496 (+ 1153us) server_negotiation.cc:407] Connection header received
0923 17:20:18.283304 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:18.283307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:18.283365 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:18.283456 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:18.284643 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.285190 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.286484 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.286721 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.286836 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:18.288100 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:18.288103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:18.288105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:18.288118 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:18.288139 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:18.288147 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:18.288222 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:18.288441 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 17:20:18.288509 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:20:18.289974 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.280624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44633 (local address 127.23.214.129:54949)
0923 17:20:18.281335 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:18.281387 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:18.283062 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:18.283726 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:18.283739 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:18.283867 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:18.284491 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:18.284528 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.285367 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.285370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:18.286358 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:18.286368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.286887 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.286892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:18.287055 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:18.287741 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:20:18.287746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:18.287805 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:20:18.287918 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:18.288243 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:18.288256 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:18.288265 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:20:18.288405 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 17:20:18.288734 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260923 17:20:18.291706 1394 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44633
I20260923 17:20:18.292029 1394 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:18.292627 1394 heartbeater.cc:511] Master 127.23.214.190:44633 requested a full tablet report, sending...
I20260923 17:20:18.294525 1206 ts_manager.cc:194] Registered new tserver with Master: 8c9d1531900249b58c17c49fedbb1309 (127.23.214.129:41347)
I20260923 17:20:18.296380 1206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.129:54949
W20260923 17:20:18.400064 1397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:18.400360 1397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:18.400399 1397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:18.404502 1397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:18.404575 1397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:18.404631 1397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:20:18.409466 1397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:18.410728 1397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:18.412096 1397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:18.421757 1403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:18.421751 1405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:18.421813 1402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:18.422076 1397 server_base.cc:1034] running on GCE node
I20260923 17:20:18.422510 1397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:18.423170 1397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:18.424365 1397 hybrid_clock.cc:648] HybridClock initialized: now 1790184018424341 us; error 34 us; skew 500 ppm
I20260923 17:20:18.426373 1397 webserver.cc:460] Webserver started at http://127.23.214.130:46019/ using document root <none> and password file <none>
I20260923 17:20:18.426990 1397 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:18.427085 1397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:18.427357 1397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:18.429531 1397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "d923672a077d46b9a93aedf448817e42"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "bb687cabc58c0672c66a07de758edc5d"
server_key_iv: "94ce9c814f0238f33cc5c0326cd1de14"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.430168 1397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "d923672a077d46b9a93aedf448817e42"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "bb687cabc58c0672c66a07de758edc5d"
server_key_iv: "94ce9c814f0238f33cc5c0326cd1de14"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.434186 1397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:20:18.436896 1411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.438459 1397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:20:18.438609 1397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "d923672a077d46b9a93aedf448817e42"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "bb687cabc58c0672c66a07de758edc5d"
server_key_iv: "94ce9c814f0238f33cc5c0326cd1de14"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.438735 1397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:18.454974 1397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:18.455727 1397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:18.455945 1397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:18.456563 1397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:18.457541 1397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:18.457613 1397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.457690 1397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:18.457731 1397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.467099 1397 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:33753
I20260923 17:20:18.467116 1524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:33753 every 8 connection(s)
I20260923 17:20:18.468101 1397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:20:18.472425 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 1397
I20260923 17:20:18.472561 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:20:18.472865 24410 external_mini_cluster.cc:1428] Setting key 91425681efa62c58ec402df45fa4f677
I20260923 17:20:18.474917 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44633
--builtin_ntp_servers=127.23.214.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:20:18.478896 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.470174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:54599 (local address 127.23.214.190:44633)
0923 17:20:18.470314 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:18.470318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:18.471434 (+ 1116us) server_negotiation.cc:407] Connection header received
0923 17:20:18.472986 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:18.472990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:18.473048 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:18.473160 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:18.474162 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.475007 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.476620 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.476830 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.476932 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:18.478306 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:18.478310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:18.478312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:18.478328 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:18.478350 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:18.478360 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:18.478439 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:18.478662 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 17:20:18.478720 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:20:18.480176 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.470584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44633 (local address 127.23.214.130:54599)
0923 17:20:18.471250 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:18.471293 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:18.472743 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:18.473346 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:18.473353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:18.473450 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:18.474025 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:18.474040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.475198 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.475200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:18.476484 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:18.476498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.477005 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.477011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:18.477258 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:18.477970 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:20:18.477975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:18.478033 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:20:18.478124 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:18.478458 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:18.478470 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:18.478479 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:20:18.478616 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:20:18.478970 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260923 17:20:18.481521 1525 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44633
I20260923 17:20:18.481777 1525 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:18.482357 1525 heartbeater.cc:511] Master 127.23.214.190:44633 requested a full tablet report, sending...
I20260923 17:20:18.483458 1206 ts_manager.cc:194] Registered new tserver with Master: d923672a077d46b9a93aedf448817e42 (127.23.214.130:33753)
I20260923 17:20:18.484047 1206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.130:54599
W20260923 17:20:18.591775 1528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:18.592088 1528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:18.592166 1528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:18.596313 1528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:18.596401 1528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:18.596498 1528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:20:18.601329 1528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:18.602533 1528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:18.603940 1528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:18.614120 1533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:18.614171 1534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:18.614120 1536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:18.614310 1528 server_base.cc:1034] running on GCE node
I20260923 17:20:18.614707 1528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:18.615350 1528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:18.616498 1528 hybrid_clock.cc:648] HybridClock initialized: now 1790184018616470 us; error 48 us; skew 500 ppm
I20260923 17:20:18.618507 1528 webserver.cc:460] Webserver started at http://127.23.214.131:43801/ using document root <none> and password file <none>
I20260923 17:20:18.619046 1528 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:18.619105 1528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:18.619270 1528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:18.621311 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "28e6cf9ede5f4a6980775e89a060b5af"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "e7a41e4970b3b900de3dd19e3893b044"
server_key_iv: "95f364a1d05618c4c7f93f9027723f3d"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.621842 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "28e6cf9ede5f4a6980775e89a060b5af"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "e7a41e4970b3b900de3dd19e3893b044"
server_key_iv: "95f364a1d05618c4c7f93f9027723f3d"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.625592 1528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:20:18.628341 1542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.629683 1528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:20:18.629824 1528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "28e6cf9ede5f4a6980775e89a060b5af"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "e7a41e4970b3b900de3dd19e3893b044"
server_key_iv: "95f364a1d05618c4c7f93f9027723f3d"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.629995 1528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:18.684443 1528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:18.685254 1528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:18.685487 1528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:18.686142 1528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:18.687144 1528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:18.687219 1528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.687294 1528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:18.687351 1528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.696306 1528 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:37381
I20260923 17:20:18.696332 1655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:37381 every 8 connection(s)
I20260923 17:20:18.697597 1528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:20:18.702952 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 1528
I20260923 17:20:18.703061 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:20:18.703354 24410 external_mini_cluster.cc:1428] Setting key cd8e34635a99932af417fbb412b99a6e
I20260923 17:20:18.708185 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.700021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:53495 (local address 127.23.214.190:44633)
0923 17:20:18.700224 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:18.700229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:18.701240 (+ 1011us) server_negotiation.cc:407] Connection header received
0923 17:20:18.702667 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:18.702670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:18.702717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:18.702780 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:18.703924 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.704741 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.706165 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.706436 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.706538 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:18.707698 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:18.707711 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:18.707713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:18.707725 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:18.707748 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:18.707756 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:18.707828 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:18.708016 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 17:20:18.708068 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:20:18.709107 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.700472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44633 (local address 127.23.214.131:53495)
0923 17:20:18.701117 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:18.701157 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:18.702481 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:18.702966 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:18.702977 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:18.703089 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:18.703751 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:18.703768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.704874 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.704876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:18.706050 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:18.706059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:18.706587 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:18.706590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:18.706734 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:18.707445 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:20:18.707446 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:18.707489 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:20:18.707555 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:18.707873 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:18.707881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:18.707886 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:20:18.708006 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 17:20:18.708282 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260923 17:20:18.710319 1656 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44633
I20260923 17:20:18.710562 1656 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:18.711122 1656 heartbeater.cc:511] Master 127.23.214.190:44633 requested a full tablet report, sending...
I20260923 17:20:18.712127 1206 ts_manager.cc:194] Registered new tserver with Master: 28e6cf9ede5f4a6980775e89a060b5af (127.23.214.131:37381)
I20260923 17:20:18.712688 1206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.131:53495
I20260923 17:20:18.717278 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:20:18.720165 1245 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:18.719256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.23.214.190:44633)
0923 17:20:18.719412 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:18.719417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:18.719579 (+ 162us) server_negotiation.cc:407] Connection header received
0923 17:20:18.719673 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:18.719675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:18.719715 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:18.719781 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:18.720030 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42830: BlockingRecv error: recv got EOF from 127.0.0.1:42830 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:20:18.720264 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 1266
I20260923 17:20:18.725749 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 1397
I20260923 17:20:18.731863 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 1528
I20260923 17:20:18.737421 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 1175
2026-09-23T17:20:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (949 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T17:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:20:18Z Disabled control of system clock
I20260923 17:20:18.768303 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:46033
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:46033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:20:18.886407 1667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:18.886675 1667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:18.886710 1667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:18.892144 1667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:20:18.892239 1667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:18.892282 1667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:18.892325 1667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:20:18.892355 1667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:20:18.901515 1667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:46033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:46033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:18.903462 1667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:18.905853 1667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:18.916159 1675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:18.916222 1667 server_base.cc:1034] running on GCE node
W20260923 17:20:18.916285 1672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:18.916440 1673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:18.916996 1667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:18.918699 1667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:18.919929 1667 hybrid_clock.cc:648] HybridClock initialized: now 1790184018919894 us; error 29 us; skew 500 ppm
I20260923 17:20:18.922297 1667 webserver.cc:460] Webserver started at http://127.23.214.190:39407/ using document root <none> and password file <none>
I20260923 17:20:18.922883 1667 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:18.922945 1667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:18.923192 1667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:18.926664 1667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "bc44421fa5e54cd8bb1f68aa948c14d3"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "2db1736d4a2c23c79af54e3b30fa5963"
server_key_iv: "0a4919a090f06e8a7b41aec9f0c334e5"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.927501 1667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "bc44421fa5e54cd8bb1f68aa948c14d3"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "2db1736d4a2c23c79af54e3b30fa5963"
server_key_iv: "0a4919a090f06e8a7b41aec9f0c334e5"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.931949 1667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:20:18.934729 1681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:18.936403 1667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:20:18.936539 1667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "bc44421fa5e54cd8bb1f68aa948c14d3"
format_stamp: "Formatted at 2026-09-23 17:20:18 on dist-test-slave-9q7f"
server_key: "2db1736d4a2c23c79af54e3b30fa5963"
server_key_iv: "0a4919a090f06e8a7b41aec9f0c334e5"
server_key_version: "encryptionkey@0"
I20260923 17:20:18.936671 1667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:18.943276 1667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:18.943981 1667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:18.944188 1667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:18.951797 1667 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:46033
I20260923 17:20:18.951790 1733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:46033 every 8 connection(s)
I20260923 17:20:18.952881 1667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:20:18.954610 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 1667
I20260923 17:20:18.954718 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:20:18.955049 24410 external_mini_cluster.cc:1428] Setting key 079b5947600609edb0df64111ad07349
I20260923 17:20:18.957163 1734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:20:18.961861 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:18.956466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.23.214.190:46033)
0923 17:20:18.957233 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:18.957256 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:18.957305 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:20:18.958342 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:18.958368 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:18.958854 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:18.959285 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:18.960172 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:18.960181 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:18.960199 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:18.960212 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:18.960286 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:18.960311 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:18.960491 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:18.960571 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 17:20:18.960859 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":121,"threads_started":1}
I20260923 17:20:18.964854 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3: Bootstrap starting.
I20260923 17:20:18.967387 1734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3: Neither blocks nor log segments found. Creating new log.
I20260923 17:20:18.968124 1734 log.cc:824] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:18.970319 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3: No bootstrap required, opened a new log
I20260923 17:20:18.974087 1734 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46033 } }
I20260923 17:20:18.974332 1734 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:20:18.974370 1734 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc44421fa5e54cd8bb1f68aa948c14d3, State: Initialized, Role: FOLLOWER
I20260923 17:20:18.974881 1734 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46033 } }
I20260923 17:20:18.975008 1734 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:20:18.975093 1734 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:20:18.975212 1734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:20:18.976418 1734 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46033 } }
I20260923 17:20:18.976794 1734 leader_election.cc:302] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc44421fa5e54cd8bb1f68aa948c14d3; no voters:
I20260923 17:20:18.977131 1734 leader_election.cc:288] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:20:18.977327 1739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:20:18.977592 1739 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [term 1 LEADER]: Becoming Leader. State: Replica: bc44421fa5e54cd8bb1f68aa948c14d3, State: Running, Role: LEADER
I20260923 17:20:18.977949 1739 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46033 } }
I20260923 17:20:18.978538 1734 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:20:18.979791 1740 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46033 } } }
I20260923 17:20:18.979935 1740 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:18.980191 1741 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc44421fa5e54cd8bb1f68aa948c14d3. Latest consensus state: current_term: 1 leader_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44421fa5e54cd8bb1f68aa948c14d3" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46033 } } }
I20260923 17:20:18.980294 1741 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:18.980408 1748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:20:18.983258 1748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:20:18.989603 1748 catalog_manager.cc:1347] Generated new cluster ID: 77130c2f263946508e1ecfc3e6ae3aff
I20260923 17:20:18.989709 1748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:20:19.002314 1748 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:20:19.002787 1748 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:20:19.010501 1748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc44421fa5e54cd8bb1f68aa948c14d3: Generated new TSK 0
I20260923 17:20:19.011264 1748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:20:19.014839 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:46033
--builtin_ntp_servers=127.23.214.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:20:19.136356 1758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:19.136641 1758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:19.136683 1758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:19.140915 1758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:19.140990 1758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:19.141052 1758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:20:19.146121 1758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:19.147554 1758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:19.149029 1758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:19.159132 1763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:19.159125 1764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:19.159125 1766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:19.159263 1758 server_base.cc:1034] running on GCE node
I20260923 17:20:19.159984 1758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:19.160617 1758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:19.161782 1758 hybrid_clock.cc:648] HybridClock initialized: now 1790184019161761 us; error 32 us; skew 500 ppm
I20260923 17:20:19.163846 1758 webserver.cc:460] Webserver started at http://127.23.214.129:38921/ using document root <none> and password file <none>
I20260923 17:20:19.164427 1758 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:19.164491 1758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:19.164660 1758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:19.166832 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "b4726b1fef5e4b9b9eac724ea0a1418c"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "5470ce8180b0a76548b8ad263148c15d"
server_key_iv: "c0c548e9d1cb695bf3a842ba7b805e30"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.167361 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "b4726b1fef5e4b9b9eac724ea0a1418c"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "5470ce8180b0a76548b8ad263148c15d"
server_key_iv: "c0c548e9d1cb695bf3a842ba7b805e30"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.171165 1758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:20:19.173817 1772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.175141 1758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:20:19.175292 1758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b4726b1fef5e4b9b9eac724ea0a1418c"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "5470ce8180b0a76548b8ad263148c15d"
server_key_iv: "c0c548e9d1cb695bf3a842ba7b805e30"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.175422 1758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:19.180447 1758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:19.181113 1758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:19.181305 1758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:19.181860 1758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:19.182850 1758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:19.182931 1758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.182997 1758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:19.183053 1758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.191957 1758 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:45257
I20260923 17:20:19.191998 1885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:45257 every 8 connection(s)
I20260923 17:20:19.192919 1758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:20:19.199538 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:19.194922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:59387 (local address 127.23.214.190:46033)
0923 17:20:19.195062 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:19.195066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:19.196306 (+ 1240us) server_negotiation.cc:407] Connection header received
0923 17:20:19.197668 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:19.197671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:19.197716 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:19.197795 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:19.199034 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:19.199037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:19.199040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:19.199053 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:19.199076 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:19.199086 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:19.199193 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:19.199351 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 17:20:19.199398 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:20:19.200470 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:19.195334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:46033 (local address 127.23.214.129:59387)
0923 17:20:19.196099 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:19.196142 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:19.197436 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:19.198016 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:19.198023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:19.198119 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:19.198756 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:20:19.198757 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:19.198807 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:20:19.198877 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:19.199187 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:19.199195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:19.199204 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:20:19.199387 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 17:20:19.199638 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":157,"threads_started":1}
I20260923 17:20:19.201601 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 1758
I20260923 17:20:19.201717 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:20:19.201815 1886 heartbeater.cc:347] Connected to a master server at 127.23.214.190:46033
I20260923 17:20:19.202093 24410 external_mini_cluster.cc:1428] Setting key 7e5ae4abaa9a8d4f6292870c1b62eb77
I20260923 17:20:19.202111 1886 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:19.202529 1886 heartbeater.cc:511] Master 127.23.214.190:46033 requested a full tablet report, sending...
I20260923 17:20:19.203958 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:46033
--builtin_ntp_servers=127.23.214.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:20:19.204030 1692 ts_manager.cc:194] Registered new tserver with Master: b4726b1fef5e4b9b9eac724ea0a1418c (127.23.214.129:45257)
W20260923 17:20:19.317087 1889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:19.317411 1889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:19.317454 1889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:19.321524 1889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:19.321580 1889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:19.321640 1889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:20:19.326496 1889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:19.327582 1889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:19.328982 1889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:19.339095 1897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:19.339077 1895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:19.339135 1889 server_base.cc:1034] running on GCE node
W20260923 17:20:19.339078 1894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:19.339787 1889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:19.340468 1889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:19.341683 1889 hybrid_clock.cc:648] HybridClock initialized: now 1790184019341664 us; error 29 us; skew 500 ppm
I20260923 17:20:19.343834 1889 webserver.cc:460] Webserver started at http://127.23.214.130:39361/ using document root <none> and password file <none>
I20260923 17:20:19.344449 1889 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:19.344543 1889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:19.344774 1889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:19.346926 1889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "d77502238a8e47e4bf4ae5e8f04a6c07"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "32c66d623c44c9eda895a4b508a77cd2"
server_key_iv: "15c10bab9ced91f465fb42eb6638ede3"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.347494 1889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "d77502238a8e47e4bf4ae5e8f04a6c07"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "32c66d623c44c9eda895a4b508a77cd2"
server_key_iv: "15c10bab9ced91f465fb42eb6638ede3"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.351174 1889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:20:19.353993 1903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.355283 1889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:20:19.355466 1889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "d77502238a8e47e4bf4ae5e8f04a6c07"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "32c66d623c44c9eda895a4b508a77cd2"
server_key_iv: "15c10bab9ced91f465fb42eb6638ede3"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.355592 1889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:19.360348 1889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:19.360971 1889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:19.361158 1889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:19.361718 1889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:19.362713 1889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:19.362787 1889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.362850 1889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:19.362900 1889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.371654 1889 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:46663
I20260923 17:20:19.371670 2016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:46663 every 8 connection(s)
I20260923 17:20:19.372639 1889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:20:19.378933 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:19.374497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:33443 (local address 127.23.214.190:46033)
0923 17:20:19.374663 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:19.374668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:19.375755 (+ 1087us) server_negotiation.cc:407] Connection header received
0923 17:20:19.377016 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:19.377019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:19.377063 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:19.377159 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:19.378366 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:19.378369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:19.378371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:19.378383 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:19.378404 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:19.378412 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:19.378568 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:19.378768 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 17:20:19.378819 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:20:19.379983 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:19.374924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:46033 (local address 127.23.214.130:33443)
0923 17:20:19.375574 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:19.375616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:19.376780 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:19.377302 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:19.377308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:19.377394 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:19.378055 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:20:19.378056 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:19.378096 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:20:19.378168 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:19.378589 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:19.378599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:19.378605 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:20:19.378762 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 17:20:19.378962 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260923 17:20:19.380708 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 1889
I20260923 17:20:19.380959 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:20:19.381330 24410 external_mini_cluster.cc:1428] Setting key 18ec4748166ee3c782bf8e9f228d56f8
I20260923 17:20:19.381326 2017 heartbeater.cc:347] Connected to a master server at 127.23.214.190:46033
I20260923 17:20:19.381567 2017 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:19.382064 2017 heartbeater.cc:511] Master 127.23.214.190:46033 requested a full tablet report, sending...
I20260923 17:20:19.383249 1692 ts_manager.cc:194] Registered new tserver with Master: d77502238a8e47e4bf4ae5e8f04a6c07 (127.23.214.130:46663)
I20260923 17:20:19.383374 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:46033
--builtin_ntp_servers=127.23.214.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:20:19.501808 2020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:19.502135 2020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:19.502205 2020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:19.506235 2020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:19.506315 2020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:19.506400 2020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:20:19.511613 2020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:19.512888 2020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:19.514323 2020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:19.524013 2026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:19.524012 2025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:19.524148 2020 server_base.cc:1034] running on GCE node
W20260923 17:20:19.524039 2028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:19.524746 2020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:19.525389 2020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:19.526574 2020 hybrid_clock.cc:648] HybridClock initialized: now 1790184019526560 us; error 31 us; skew 500 ppm
I20260923 17:20:19.528461 2020 webserver.cc:460] Webserver started at http://127.23.214.131:41215/ using document root <none> and password file <none>
I20260923 17:20:19.529062 2020 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:19.529155 2020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:19.529379 2020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:19.531608 2020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "bcca3ecbc18b4e99a9c25a698c07b3fa"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "8a1d3e0cd1da406c9fe3213688d4d835"
server_key_iv: "9903ac44088f6639c88d34fc638e7d2c"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.532186 2020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "bcca3ecbc18b4e99a9c25a698c07b3fa"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "8a1d3e0cd1da406c9fe3213688d4d835"
server_key_iv: "9903ac44088f6639c88d34fc638e7d2c"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.536036 2020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260923 17:20:19.538645 2034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.539933 2020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:20:19.540066 2020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "bcca3ecbc18b4e99a9c25a698c07b3fa"
format_stamp: "Formatted at 2026-09-23 17:20:19 on dist-test-slave-9q7f"
server_key: "8a1d3e0cd1da406c9fe3213688d4d835"
server_key_iv: "9903ac44088f6639c88d34fc638e7d2c"
server_key_version: "encryptionkey@0"
I20260923 17:20:19.540180 2020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:19.544765 2020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:19.545392 2020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:19.545568 2020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:19.546175 2020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:19.547076 2020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:19.547143 2020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.547209 2020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:19.547257 2020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:19.556118 2020 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:45655
I20260923 17:20:19.556139 2147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:45655 every 8 connection(s)
I20260923 17:20:19.557119 2020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:20:19.560458 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 2020
I20260923 17:20:19.560581 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:20:19.560915 24410 external_mini_cluster.cc:1428] Setting key a0371426fbf06a46b5c90b1ca2fef21f
I20260923 17:20:19.564003 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:19.559084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:33787 (local address 127.23.214.190:46033)
0923 17:20:19.559230 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:19.559233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:19.560555 (+ 1322us) server_negotiation.cc:407] Connection header received
0923 17:20:19.562145 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:19.562148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:19.562197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:19.562284 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:19.563519 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:19.563522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:19.563525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:20:19.563538 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:19.563562 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:20:19.563571 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:19.563672 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:19.563822 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:20:19.563868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:20:19.564950 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:19.559546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:46033 (local address 127.23.214.131:33787)
0923 17:20:19.560313 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:19.560411 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:19.561881 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:19.562490 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:19.562498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:19.562614 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:19.563246 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:20:19.563248 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:19.563300 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:20:19.563373 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:19.563662 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:19.563671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:19.563679 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:20:19.563888 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 17:20:19.564144 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":111,"threads_started":1}
I20260923 17:20:19.566102 2148 heartbeater.cc:347] Connected to a master server at 127.23.214.190:46033
I20260923 17:20:19.566331 2148 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:19.566762 2148 heartbeater.cc:511] Master 127.23.214.190:46033 requested a full tablet report, sending...
I20260923 17:20:19.567863 1692 ts_manager.cc:194] Registered new tserver with Master: bcca3ecbc18b4e99a9c25a698c07b3fa (127.23.214.131:45655)
I20260923 17:20:19.574931 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:20:19.578004 1737 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:20:19.576957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.23.214.190:46033)
0923 17:20:19.577128 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:19.577135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:19.577308 (+ 173us) server_negotiation.cc:407] Connection header received
0923 17:20:19.577447 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:19.577450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:19.577493 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:19.577577 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:19.577848 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58868: BlockingRecv error: recv got EOF from 127.0.0.1:58868 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:20:19.578544 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 1758
I20260923 17:20:19.584316 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 1889
I20260923 17:20:19.590082 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 2020
I20260923 17:20:19.595469 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 1667
2026-09-23T17:20:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (856 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:20:19 dist-test-slave-9q7f krb5kdc[2158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:20:19 dist-test-slave-9q7f krb5kdc[2158](info): set up 2 sockets
Sep 23 17:20:19 dist-test-slave-9q7f krb5kdc[2158](info): commencing operation
krb5kdc: starting...
W20260923 17:20:20.641112 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:20:20 dist-test-slave-9q7f krb5kdc[2158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:20:20.769770 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38657
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:38657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:20.884562 2171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:20.884925 2171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:20.885001 2171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:20.889312 2171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:20:20.889443 2171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:20.889495 2171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:20.889556 2171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:20:20.889596 2171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:20:20.895273 2171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:38657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:20.896534 2171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:20.897665 2171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:20.906629 2177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:20.906641 2179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:20.906643 2176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:20.906695 2171 server_base.cc:1034] running on GCE node
I20260923 17:20:20.907473 2171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:20.908666 2171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:20.909919 2171 hybrid_clock.cc:648] HybridClock initialized: now 1790184020909855 us; error 29 us; skew 500 ppm
Sep 23 17:20:20 dist-test-slave-9q7f krb5kdc[2158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:20.912824 2171 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:20:20.913785 2171 webserver.cc:460] Webserver started at http://127.23.214.190:34559/ using document root <none> and password file <none>
I20260923 17:20:20.914434 2171 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:20.914529 2171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:20.914762 2171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:20.916908 2171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "8adfdb1a49a94138a16131aef38efd2d"
format_stamp: "Formatted at 2026-09-23 17:20:20 on dist-test-slave-9q7f"
server_key: "b86e6e2d67d3b65f7bc3ad3846853789"
server_key_iv: "5a4498ac11822825db1c79e1d1675197"
server_key_version: "encryptionkey@0"
I20260923 17:20:20.917529 2171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "8adfdb1a49a94138a16131aef38efd2d"
format_stamp: "Formatted at 2026-09-23 17:20:20 on dist-test-slave-9q7f"
server_key: "b86e6e2d67d3b65f7bc3ad3846853789"
server_key_iv: "5a4498ac11822825db1c79e1d1675197"
server_key_version: "encryptionkey@0"
I20260923 17:20:20.921533 2171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 17:20:20.924142 2186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:20.925434 2171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:20:20.925583 2171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "8adfdb1a49a94138a16131aef38efd2d"
format_stamp: "Formatted at 2026-09-23 17:20:20 on dist-test-slave-9q7f"
server_key: "b86e6e2d67d3b65f7bc3ad3846853789"
server_key_iv: "5a4498ac11822825db1c79e1d1675197"
server_key_version: "encryptionkey@0"
I20260923 17:20:20.925707 2171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:20.949684 2171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:20.953477 2171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:20.953711 2171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:20.961439 2171 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:38657
I20260923 17:20:20.961459 2238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:38657 every 8 connection(s)
I20260923 17:20:20.962527 2171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:20:20.966753 2239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:20:20.966928 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 2171
I20260923 17:20:20.967056 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:20:20.967388 24410 external_mini_cluster.cc:1428] Setting key 924444074df99c7551e987126caf1da3
I20260923 17:20:20.975425 2239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d: Bootstrap starting.
Sep 23 17:20:20 dist-test-slave-9q7f krb5kdc[2158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:20:20.978588 2239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d: Neither blocks nor log segments found. Creating new log.
I20260923 17:20:20.979449 2239 log.cc:824] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:20.981976 2239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d: No bootstrap required, opened a new log
I20260923 17:20:20.983915 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:20.969490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.23.214.190:38657)
0923 17:20:20.970059 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:20.970076 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:20.970117 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:20:20.970758 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:20.970779 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:20.971153 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:20.971479 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:20.972536 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:20.973609 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:20.974832 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:20.975336 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:20.975475 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:20.978541 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:20.978547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:20.978562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:20.978596 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:20.980721 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:20.981245 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:20.981250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:20.981254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:20.981380 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:20.981684 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:20.981686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:20.981687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:20.982213 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:20.982370 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:20.982738 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 17:20:20.983070 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":195,"threads_started":1}
I20260923 17:20:20.986509 2239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adfdb1a49a94138a16131aef38efd2d" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38657 } }
I20260923 17:20:20.986768 2239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:20:20.986872 2239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adfdb1a49a94138a16131aef38efd2d, State: Initialized, Role: FOLLOWER
I20260923 17:20:20.987323 2239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adfdb1a49a94138a16131aef38efd2d" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38657 } }
I20260923 17:20:20.987480 2239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:20:20.987566 2239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:20:20.987682 2239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:20:20.988776 2239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adfdb1a49a94138a16131aef38efd2d" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38657 } }
I20260923 17:20:20.989128 2239 leader_election.cc:302] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8adfdb1a49a94138a16131aef38efd2d; no voters:
I20260923 17:20:20.989457 2239 leader_election.cc:288] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:20:20.989607 2244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:20:20.989857 2244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [term 1 LEADER]: Becoming Leader. State: Replica: 8adfdb1a49a94138a16131aef38efd2d, State: Running, Role: LEADER
I20260923 17:20:20.990211 2244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adfdb1a49a94138a16131aef38efd2d" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38657 } }
I20260923 17:20:20.990867 2239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:20:20.991914 2245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8adfdb1a49a94138a16131aef38efd2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adfdb1a49a94138a16131aef38efd2d" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38657 } } }
I20260923 17:20:20.992041 2245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:20.992336 2246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8adfdb1a49a94138a16131aef38efd2d. Latest consensus state: current_term: 1 leader_uuid: "8adfdb1a49a94138a16131aef38efd2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adfdb1a49a94138a16131aef38efd2d" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38657 } } }
I20260923 17:20:20.992430 2246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d [sys.catalog]: This master's current role is: LEADER
I20260923 17:20:20.992443 2253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:20:20.995429 2253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:20:21.001034 2253 catalog_manager.cc:1347] Generated new cluster ID: afd6c0154a244f2dba797fb23b7b8b27
I20260923 17:20:21.001106 2253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:20:21.011355 2253 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:20:21.012676 2253 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:20:21.022307 2253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d: Generated new TSK 0
I20260923 17:20:21.023170 2253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:20:21.061020 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38657
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:21.179217 2267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:21.179483 2267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:21.179526 2267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:21.183493 2267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:21.183555 2267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:21.183615 2267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:20:21.189025 2267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:21.190166 2267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:21.191216 2267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:20:21.201460 2267 server_base.cc:1034] running on GCE node
W20260923 17:20:21.201406 2273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:21.201406 2272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:21.201445 2275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:21.202183 2267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:21.202889 2267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:21.204064 2267 hybrid_clock.cc:648] HybridClock initialized: now 1790184021204045 us; error 35 us; skew 500 ppm
Sep 23 17:20:21 dist-test-slave-9q7f krb5kdc[2158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:21.207177 2267 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:20:21.208061 2267 webserver.cc:460] Webserver started at http://127.23.214.129:35025/ using document root <none> and password file <none>
I20260923 17:20:21.208608 2267 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:21.208667 2267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:21.208833 2267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:21.211001 2267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "981d09ef4d4a4546a03864b7e577731c"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "9af8c345954aa32cb5d922ff774c37b2"
server_key_iv: "071e4b9677e344a4dd438ec665681cb2"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.211539 2267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "981d09ef4d4a4546a03864b7e577731c"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "9af8c345954aa32cb5d922ff774c37b2"
server_key_iv: "071e4b9677e344a4dd438ec665681cb2"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.215530 2267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:20:21.218135 2283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.219422 2267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:20:21.219551 2267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "981d09ef4d4a4546a03864b7e577731c"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "9af8c345954aa32cb5d922ff774c37b2"
server_key_iv: "071e4b9677e344a4dd438ec665681cb2"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.219663 2267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:21.244454 2267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:21.248011 2267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:21.248235 2267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:21.248862 2267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:21.249768 2267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:21.249814 2267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.249857 2267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:21.249931 2267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.258973 2267 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:43669
I20260923 17:20:21.259014 2400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:43669 every 8 connection(s)
I20260923 17:20:21.259997 2267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:20:21.268462 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 2267
I20260923 17:20:21.268607 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:20:21.268945 24410 external_mini_cluster.cc:1428] Setting key b0d2e96fbf6089069ff308d55d661d98
Sep 23 17:20:21 dist-test-slave-9q7f krb5kdc[2158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:20:21.275373 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.261985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:42625 (local address 127.23.214.190:38657)
0923 17:20:21.262144 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:21.262149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:21.263008 (+ 859us) server_negotiation.cc:407] Connection header received
0923 17:20:21.264393 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:21.264396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:21.264444 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:21.264520 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:21.265741 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.266302 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.267520 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.267761 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.267849 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.271084 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:21.271087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:21.271089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:21.271112 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:21.272962 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.273506 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.273512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.273517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.273617 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.274081 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.274084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.274085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.274267 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:21.274346 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:21.275121 (+ 775us) server_negotiation.cc:299] Negotiation successful
0923 17:20:21.275247 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:20:21.276669 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.262400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38657 (local address 127.23.214.129:42625)
0923 17:20:21.262840 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:21.262895 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:21.264159 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:21.264681 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:21.264687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:21.265071 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:21.265579 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.265592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.266476 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.266480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.267404 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.267414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.267912 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.267914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.268052 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.268831 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:21.268862 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:21.270901 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:21.273104 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:21.273110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:21.273124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:21.273391 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:21.273747 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:21.273753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:21.273758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:21.273981 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:21.274375 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:21.274386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:21.274780 (+ 394us) client_negotiation.cc:769] Sending connection context
0923 17:20:21.275010 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 17:20:21.275436 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20260923 17:20:21.278087 2401 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38657
I20260923 17:20:21.278354 2401 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:21.278930 2401 heartbeater.cc:511] Master 127.23.214.190:38657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:20:21.280572 2203 ts_manager.cc:194] Registered new tserver with Master: 981d09ef4d4a4546a03864b7e577731c (127.23.214.129:43669)
I20260923 17:20:21.281852 2203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:42625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:20:21.298448 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38657
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:21.411916 2406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:21.412199 2406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:21.412245 2406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:21.416363 2406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:21.416451 2406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:21.416525 2406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:20:21.422017 2406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:21.423187 2406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:21.424332 2406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:21.434998 2412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:21.435112 2406 server_base.cc:1034] running on GCE node
W20260923 17:20:21.435000 2411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:21.435325 2414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:21.435734 2406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:21.436444 2406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:21.437616 2406 hybrid_clock.cc:648] HybridClock initialized: now 1790184021437597 us; error 30 us; skew 500 ppm
Sep 23 17:20:21 dist-test-slave-9q7f krb5kdc[2158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:21.440640 2406 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:20:21.441568 2406 webserver.cc:460] Webserver started at http://127.23.214.130:39087/ using document root <none> and password file <none>
I20260923 17:20:21.442190 2406 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:21.442253 2406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:21.442437 2406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:21.444607 2406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "062464d973e7415585cc725a09d4e63a"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "ac2bbf14e67745b7638346880c5651bd"
server_key_iv: "da07541291f658ac4ab80bed392c6367"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.445120 2406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "062464d973e7415585cc725a09d4e63a"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "ac2bbf14e67745b7638346880c5651bd"
server_key_iv: "da07541291f658ac4ab80bed392c6367"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.449018 2406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:20:21.451740 2421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.453188 2406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:20:21.453387 2406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "062464d973e7415585cc725a09d4e63a"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "ac2bbf14e67745b7638346880c5651bd"
server_key_iv: "da07541291f658ac4ab80bed392c6367"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.453521 2406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:21.476743 2406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:21.480145 2406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:21.480397 2406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:21.481025 2406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:21.481804 2406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:21.481858 2406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.481966 2406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:21.481990 2406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.491132 2406 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:37443
I20260923 17:20:21.492161 2406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:20:21.492705 2534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:37443 every 8 connection(s)
I20260923 17:20:21.495186 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 2406
I20260923 17:20:21.495290 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:20:21.495641 24410 external_mini_cluster.cc:1428] Setting key 8601953ecc5d6f9d49a96ca2267c7b97
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Sep 23 17:20:21 dist-test-slave-9q7f krb5kdc[2158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:20:21.513825 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.495269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:48035 (local address 127.23.214.190:38657)
0923 17:20:21.495441 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:21.495446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:21.495464 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:20:21.496923 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:21.496926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:21.496976 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:21.497065 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:21.498366 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.499133 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.500645 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.500873 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.500957 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.509571 (+ 8614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:21.509575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:21.509577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:21.509604 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:21.511345 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.512062 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.512065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.512066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.512155 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.512578 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.512581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.512582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.512768 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:21.512847 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:21.513581 (+ 734us) server_negotiation.cc:299] Negotiation successful
0923 17:20:21.513697 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:20:21.515157 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.494487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38657 (local address 127.23.214.130:48035)
0923 17:20:21.494956 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:21.495000 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:21.496666 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:21.497218 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:21.497224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:21.497646 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:21.498223 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.498238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.499264 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.499267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.500515 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.500529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.501036 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.501043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.501274 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.507559 (+ 6285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:21.507600 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:21.509329 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:21.511484 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:21.511493 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:21.511516 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:21.511938 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:21.512258 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:21.512262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:21.512266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:21.512478 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:21.512887 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:21.512896 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:21.513293 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 17:20:21.513555 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 17:20:21.514004 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260923 17:20:21.516435 2535 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38657
I20260923 17:20:21.516695 2535 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:21.517213 2535 heartbeater.cc:511] Master 127.23.214.190:38657 requested a full tablet report, sending...
I20260923 17:20:21.518458 2203 ts_manager.cc:194] Registered new tserver with Master: 062464d973e7415585cc725a09d4e63a (127.23.214.130:37443)
I20260923 17:20:21.519188 2203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:48035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:20:21.525413 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38657
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:36211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:20:21.639645 2540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:20:21.639933 2540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:20:21.640033 2540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:20:21.644390 2540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:20:21.644462 2540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:20:21.644567 2540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:20:21.650084 2540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:36211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:20:21.651296 2540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:20:21.652379 2540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:20:21.662308 2545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:20:21.662317 2548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:21.662441 2540 server_base.cc:1034] running on GCE node
W20260923 17:20:21.662612 2546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:20:21.663023 2540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:20:21.663690 2540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:20:21.664886 2540 hybrid_clock.cc:648] HybridClock initialized: now 1790184021664864 us; error 32 us; skew 500 ppm
Sep 23 17:20:21 dist-test-slave-9q7f krb5kdc[2158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:20:21.667802 2540 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:20:21.668759 2540 webserver.cc:460] Webserver started at http://127.23.214.131:43705/ using document root <none> and password file <none>
I20260923 17:20:21.669301 2540 fs_manager.cc:359] Metadata directory not provided
I20260923 17:20:21.669368 2540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:20:21.669529 2540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:20:21.671660 2540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "4408553f3dc147649468378a4c57900f"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "eb8d3ac472fe9608f9ce99684bc1d57b"
server_key_iv: "995f64faf395cae61feb78d42c56fc06"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.672227 2540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "4408553f3dc147649468378a4c57900f"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "eb8d3ac472fe9608f9ce99684bc1d57b"
server_key_iv: "995f64faf395cae61feb78d42c56fc06"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.675972 2540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260923 17:20:21.678503 2555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.679764 2540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:20:21.679862 2540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "4408553f3dc147649468378a4c57900f"
format_stamp: "Formatted at 2026-09-23 17:20:21 on dist-test-slave-9q7f"
server_key: "eb8d3ac472fe9608f9ce99684bc1d57b"
server_key_iv: "995f64faf395cae61feb78d42c56fc06"
server_key_version: "encryptionkey@0"
I20260923 17:20:21.680006 2540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:20:21.705739 2540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:20:21.709445 2540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:20:21.709679 2540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:20:21.710470 2540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:20:21.711548 2540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:20:21.711599 2540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.711675 2540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:20:21.711714 2540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:20:21.722524 2540 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:37937
I20260923 17:20:21.722587 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:37937 every 8 connection(s)
I20260923 17:20:21.723572 2540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:20:21.732123 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 2540
I20260923 17:20:21.732304 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:20:21.732628 24410 external_mini_cluster.cc:1428] Setting key c1a710ee58d4bc22d3e4b34261ebff51
Sep 23 17:20:21 dist-test-slave-9q7f krb5kdc[2158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184021, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:20:21.738596 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.725671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:38333 (local address 127.23.214.190:38657)
0923 17:20:21.725810 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:21.725815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:21.726759 (+ 944us) server_negotiation.cc:407] Connection header received
0923 17:20:21.728367 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:21.728371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:21.728438 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:21.728562 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:21.729854 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.730504 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.731783 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.732049 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.732179 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.735101 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:21.735104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:21.735106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:21.735131 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:21.736564 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.737078 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.737080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.737082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.737154 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.737517 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.737519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.737520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.737692 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:21.737768 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:21.738375 (+ 607us) server_negotiation.cc:299] Negotiation successful
0923 17:20:21.738476 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:20:21.739495 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.726091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38657 (local address 127.23.214.131:38333)
0923 17:20:21.726590 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:21.726637 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:21.728113 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:21.728741 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:21.728749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:21.729172 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:20:21.729688 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.729702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.730671 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.730674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.731646 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.731655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.732221 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.732226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.732460 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.733127 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:20:21.733145 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:20:21.734910 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:20:21.736685 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:21.736689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:21.736703 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:21.736982 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:21.737236 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:20:21.737238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:20:21.737240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:20:21.737427 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:20:21.737787 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:20:21.737792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:20:21.738119 (+ 327us) client_negotiation.cc:769] Sending connection context
0923 17:20:21.738338 (+ 219us) client_negotiation.cc:240] Negotiation successful
0923 17:20:21.738624 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
I20260923 17:20:21.740657 2670 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38657
I20260923 17:20:21.740895 2670 heartbeater.cc:464] Registering TS with master...
I20260923 17:20:21.741370 2670 heartbeater.cc:511] Master 127.23.214.190:38657 requested a full tablet report, sending...
I20260923 17:20:21.742369 2203 ts_manager.cc:194] Registered new tserver with Master: 4408553f3dc147649468378a4c57900f (127.23.214.131:37937)
I20260923 17:20:21.742933 2203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:38333
I20260923 17:20:21.746909 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:20:21.757570 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.749028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.23.214.190:38657)
0923 17:20:21.749222 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:21.749231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:21.749364 (+ 133us) server_negotiation.cc:407] Connection header received
0923 17:20:21.749529 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:21.749532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:21.749583 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:21.749677 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:20:21.750546 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.751109 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.752055 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.752296 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.752423 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.754400 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:20:21.754403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:20:21.754406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:20:21.754431 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:20:21.755678 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.756149 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.756154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.756156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.756253 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:20:21.756579 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:20:21.756584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:20:21.756586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:20:21.756824 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:20:21.756985 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:21.757249 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 17:20:21.757386 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 17:20:21.760622 2203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:20:21.763830 2203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:20:21.778733 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.772433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37937 (local address 127.0.0.1:53844)
0923 17:20:21.773309 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:21.773360 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:21.773488 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:21.774290 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:21.774295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:21.774315 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:20:21.774421 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.774426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.775736 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.775739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.777160 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.777168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.778150 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.778153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.778265 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.778315 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 17:20:21.778449 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 17:20:21.778572 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":26,"thread_start_us":100,"threads_started":1}
I20260923 17:20:21.778738 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.772898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.23.214.131:37937)
0923 17:20:21.773399 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:21.773415 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:21.773442 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:20:21.773690 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:21.773696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:21.773952 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:21.774085 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:20:21.774536 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.775592 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.777279 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.778028 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.778165 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.778302 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:21.778480 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 17:20:21.778572 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":116,"threads_started":1}
I20260923 17:20:21.779332 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.772645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:37443 (local address 127.0.0.1:45098)
0923 17:20:21.773660 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:21.773694 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:21.773802 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:21.774562 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:21.774565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:21.774582 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:20:21.774707 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.774714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.775969 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.775976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.777371 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.777382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.778822 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.778826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.778974 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.779000 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:20:21.779077 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:20:21.779154 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260923 17:20:21.779592 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.772865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43669 (local address 127.0.0.1:53406)
0923 17:20:21.773770 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:21.773783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:21.773975 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:21.774703 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:21.774706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:21.774721 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:20:21.774845 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.774851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.776179 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.776182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.777496 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:21.777506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.778988 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.778992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:21.779115 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.779139 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:20:21.779216 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:20:21.779280 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":8,"thread_start_us":89,"threads_started":1}
I20260923 17:20:21.779968 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.773280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.23.214.129:43669)
0923 17:20:21.773616 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:21.773630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:21.773867 (+ 237us) server_negotiation.cc:407] Connection header received
0923 17:20:21.774137 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:21.774142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:21.774356 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:21.774496 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:20:21.774972 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.776046 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.777633 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.778507 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.778624 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.778751 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:21.779264 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 17:20:21.779375 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":85,"threads_started":1}
I20260923 17:20:21.780104 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:21.773140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.23.214.130:37443)
0923 17:20:21.774142 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:21.774154 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:21.774175 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:20:21.774241 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:21.774246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:21.774400 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:21.774875 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:20:21.775047 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.775811 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.777531 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:21.778491 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:21.778624 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:21.778751 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:21.779794 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0923 17:20:21.779933 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":139,"threads_started":1}
I20260923 17:20:21.782073 2604 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5130de7054493ea0361a28ea994d2c (DEFAULT_TABLE table=test-table [id=ef34c505c413458db9dab605a7949a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:20:21.782483 2334 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5130de7054493ea0361a28ea994d2c (DEFAULT_TABLE table=test-table [id=ef34c505c413458db9dab605a7949a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:20:21.782634 2469 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5130de7054493ea0361a28ea994d2c (DEFAULT_TABLE table=test-table [id=ef34c505c413458db9dab605a7949a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:20:21.783324 2604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5130de7054493ea0361a28ea994d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:20:21.783423 2334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5130de7054493ea0361a28ea994d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:20:21.783766 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5130de7054493ea0361a28ea994d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:20:21.791129 2689 tablet_bootstrap.cc:492] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Bootstrap starting.
I20260923 17:20:21.791703 2690 tablet_bootstrap.cc:492] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: Bootstrap starting.
I20260923 17:20:21.792171 2688 tablet_bootstrap.cc:492] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: Bootstrap starting.
I20260923 17:20:21.793799 2690 tablet_bootstrap.cc:654] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: Neither blocks nor log segments found. Creating new log.
I20260923 17:20:21.794497 2689 tablet_bootstrap.cc:654] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Neither blocks nor log segments found. Creating new log.
I20260923 17:20:21.794617 2690 log.cc:824] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:21.794818 2688 tablet_bootstrap.cc:654] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: Neither blocks nor log segments found. Creating new log.
I20260923 17:20:21.795553 2689 log.cc:824] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:21.795616 2688 log.cc:824] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:20:21.796875 2690 tablet_bootstrap.cc:492] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: No bootstrap required, opened a new log
I20260923 17:20:21.797078 2690 ts_tablet_manager.cc:1397] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:20:21.797613 2688 tablet_bootstrap.cc:492] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: No bootstrap required, opened a new log
I20260923 17:20:21.797617 2689 tablet_bootstrap.cc:492] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: No bootstrap required, opened a new log
I20260923 17:20:21.797780 2688 ts_tablet_manager.cc:1397] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:20:21.797811 2689 ts_tablet_manager.cc:1397] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:20:21.801265 2689 raft_consensus.cc:348] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:21.801550 2689 raft_consensus.cc:374] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:20:21.801610 2689 raft_consensus.cc:729] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4408553f3dc147649468378a4c57900f, State: Initialized, Role: FOLLOWER
I20260923 17:20:21.801712 2690 raft_consensus.cc:348] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:21.801766 2688 raft_consensus.cc:348] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:21.802006 2688 raft_consensus.cc:374] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:20:21.802001 2690 raft_consensus.cc:374] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:20:21.802060 2688 raft_consensus.cc:729] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062464d973e7415585cc725a09d4e63a, State: Initialized, Role: FOLLOWER
I20260923 17:20:21.802060 2690 raft_consensus.cc:729] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981d09ef4d4a4546a03864b7e577731c, State: Initialized, Role: FOLLOWER
I20260923 17:20:21.802186 2689 consensus_queue.cc:260] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:21.802496 2688 consensus_queue.cc:260] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:21.802496 2690 consensus_queue.cc:260] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:21.802978 2670 heartbeater.cc:503] Master 127.23.214.190:38657 was elected leader, sending a full tablet report...
I20260923 17:20:21.803493 2690 ts_tablet_manager.cc:1428] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:20:21.803493 2688 ts_tablet_manager.cc:1428] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:20:21.803853 2401 heartbeater.cc:503] Master 127.23.214.190:38657 was elected leader, sending a full tablet report...
I20260923 17:20:21.804536 2689 ts_tablet_manager.cc:1428] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:20:21.804872 2535 heartbeater.cc:503] Master 127.23.214.190:38657 was elected leader, sending a full tablet report...
W20260923 17:20:21.974633 2671 tablet.cc:2369] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:20:21.993453 2536 tablet.cc:2369] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:20:22.005105 2694 raft_consensus.cc:482] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:20:22.005330 2694 raft_consensus.cc:504] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:22.006632 2694 leader_election.cc:288] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 062464d973e7415585cc725a09d4e63a (127.23.214.130:37443), 981d09ef4d4a4546a03864b7e577731c (127.23.214.129:43669)
I20260923 17:20:22.010854 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:22.007001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:58925 (local address 127.23.214.129:43669)
0923 17:20:22.007136 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:22.007142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:22.007161 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:20:22.007337 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:22.007340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:22.007386 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:22.007481 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:20:22.007758 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.008255 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.009565 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.010318 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.010416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:22.010460 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:22.010673 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 17:20:22.010733 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:20:22.010870 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:22.006885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43669 (local address 127.23.214.131:58925)
0923 17:20:22.007037 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:22.007057 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:22.007177 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:22.007507 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:22.007510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:22.007529 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:20:22.007646 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:22.007651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.008371 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.008378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:22.009428 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:22.009438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.010429 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.010432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:22.010541 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:22.010566 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:20:22.010647 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:20:22.010715 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 17:20:22.011368 2402 tablet.cc:2369] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:20:22.011538 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:22.006885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:37443 (local address 127.23.214.131:33025)
0923 17:20:22.007475 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:22.007487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:20:22.007612 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:20:22.007907 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:20:22.007910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:20:22.007927 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:20:22.008058 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:22.008064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.008847 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.008853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:22.010094 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:20:22.010104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.011103 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.011107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:20:22.011233 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:22.011255 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:20:22.011336 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:20:22.011408 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":184,"thread_start_us":140,"threads_started":1}
I20260923 17:20:22.011497 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5130de7054493ea0361a28ea994d2c" candidate_uuid: "4408553f3dc147649468378a4c57900f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981d09ef4d4a4546a03864b7e577731c" is_pre_election: true
I20260923 17:20:22.011585 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:22.006998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:33025 (local address 127.23.214.130:37443)
0923 17:20:22.007136 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:22.007142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:22.007577 (+ 435us) server_negotiation.cc:407] Connection header received
0923 17:20:22.007734 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:22.007737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:22.007802 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:22.007899 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:20:22.008177 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.008715 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.010221 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.010857 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.010971 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:22.011024 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:22.011374 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 17:20:22.011453 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:20:22.011858 2355 raft_consensus.cc:2454] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4408553f3dc147649468378a4c57900f in term 0.
I20260923 17:20:22.012413 2559 leader_election.cc:302] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4408553f3dc147649468378a4c57900f, 981d09ef4d4a4546a03864b7e577731c; no voters:
I20260923 17:20:22.012699 2694 raft_consensus.cc:2781] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:20:22.012673 2489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5130de7054493ea0361a28ea994d2c" candidate_uuid: "4408553f3dc147649468378a4c57900f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062464d973e7415585cc725a09d4e63a" is_pre_election: true
I20260923 17:20:22.012797 2694 raft_consensus.cc:482] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:20:22.012833 2694 raft_consensus.cc:3037] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:20:22.012984 2489 raft_consensus.cc:2454] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4408553f3dc147649468378a4c57900f in term 0.
I20260923 17:20:22.014122 2694 raft_consensus.cc:504] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:22.014513 2694 leader_election.cc:288] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [CANDIDATE]: Term 1 election: Requested vote from peers 062464d973e7415585cc725a09d4e63a (127.23.214.130:37443), 981d09ef4d4a4546a03864b7e577731c (127.23.214.129:43669)
I20260923 17:20:22.014860 2489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5130de7054493ea0361a28ea994d2c" candidate_uuid: "4408553f3dc147649468378a4c57900f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062464d973e7415585cc725a09d4e63a"
I20260923 17:20:22.014937 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5130de7054493ea0361a28ea994d2c" candidate_uuid: "4408553f3dc147649468378a4c57900f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981d09ef4d4a4546a03864b7e577731c"
I20260923 17:20:22.014992 2489 raft_consensus.cc:3037] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:20:22.015053 2355 raft_consensus.cc:3037] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:20:22.016307 2489 raft_consensus.cc:2454] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4408553f3dc147649468378a4c57900f in term 1.
I20260923 17:20:22.016307 2355 raft_consensus.cc:2454] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4408553f3dc147649468378a4c57900f in term 1.
I20260923 17:20:22.016587 2556 leader_election.cc:302] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062464d973e7415585cc725a09d4e63a, 4408553f3dc147649468378a4c57900f; no voters:
I20260923 17:20:22.016757 2694 raft_consensus.cc:2781] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:20:22.016996 2694 raft_consensus.cc:686] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 1 LEADER]: Becoming Leader. State: Replica: 4408553f3dc147649468378a4c57900f, State: Running, Role: LEADER
I20260923 17:20:22.017326 2694 consensus_queue.cc:237] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } }
I20260923 17:20:22.020493 2201 catalog_manager.cc:5495] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4408553f3dc147649468378a4c57900f (127.23.214.131). New cstate: current_term: 1 leader_uuid: "4408553f3dc147649468378a4c57900f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408553f3dc147649468378a4c57900f" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 } health_report { overall_health: UNKNOWN } } }
I20260923 17:20:22.076761 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:22.072963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.23.214.131:37937)
0923 17:20:22.073106 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 17:20:22.073111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:20:22.073127 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:20:22.073252 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:20:22.073254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:20:22.073311 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:20:22.073432 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:20:22.073678 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.074258 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.075170 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:20:22.075397 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:20:22.075515 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:20:22.075834 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:20:22.076301 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:20:22.076450 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:20:22.076538 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 17:20:22.076606 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:20:22.082800 2355 raft_consensus.cc:1261] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c [term 1 FOLLOWER]: Refusing update from remote peer 4408553f3dc147649468378a4c57900f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:20:22.082800 2489 raft_consensus.cc:1261] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 1 FOLLOWER]: Refusing update from remote peer 4408553f3dc147649468378a4c57900f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:20:22.083628 2698 consensus_queue.cc:1035] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [LEADER]: Connected to new peer: Peer: permanent_uuid: "981d09ef4d4a4546a03864b7e577731c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:20:22.083832 2694 consensus_queue.cc:1035] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [LEADER]: Connected to new peer: Peer: permanent_uuid: "062464d973e7415585cc725a09d4e63a" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:20:22.090729 2701 mvcc.cc:205] Tried to move back new op lower bound from 7332593754443366400 to 7332593754185883648. Current Snapshot: MvccSnapshot[applied={T|T < 7332593754443366400}]
I20260923 17:20:22.091337 2703 mvcc.cc:205] Tried to move back new op lower bound from 7332593754443366400 to 7332593754185883648. Current Snapshot: MvccSnapshot[applied={T|T < 7332593754443366400}]
I20260923 17:20:22.092819 2702 mvcc.cc:205] Tried to move back new op lower bound from 7332593754443366400 to 7332593754185883648. Current Snapshot: MvccSnapshot[applied={T|T < 7332593754443366400}]
I20260923 17:20:22.100235 2202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:20:22.100442 2202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:20:22.103210 2202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8adfdb1a49a94138a16131aef38efd2d: Sending DeleteTablet for 3 replicas of tablet 5d5130de7054493ea0361a28ea994d2c
I20260923 17:20:22.104100 2604 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5130de7054493ea0361a28ea994d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:20:22 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:53844
I20260923 17:20:22.104104 2469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5130de7054493ea0361a28ea994d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:20:22 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:45098
I20260923 17:20:22.104287 2334 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5130de7054493ea0361a28ea994d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:20:22 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:53406
I20260923 17:20:22.104976 2712 tablet_replica.cc:330] T 5d5130de7054493ea0361a28ea994d2c P 981d09ef4d4a4546a03864b7e577731c: stopping tablet replica
I20260923 17:20:22.105100 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 2267
I20260923 17:20:22.110145 2711 tablet_replica.cc:330] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: stopping tablet replica
I20260923 17:20:22.110183 2710 tablet_replica.cc:330] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: stopping tablet replica
I20260923 17:20:22.110575 2710 raft_consensus.cc:2229] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:20:22.110611 2711 raft_consensus.cc:2229] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:20:22.111056 2710 raft_consensus.cc:2258] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:20:22.111069 2711 raft_consensus.cc:2258] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:20:22.112824 2711 ts_tablet_manager.cc:1905] T 5d5130de7054493ea0361a28ea994d2c P 062464d973e7415585cc725a09d4e63a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:20:22.113117 2710 ts_tablet_manager.cc:1905] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 17:20:22.113736 2190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:43669 (error 108)
I20260923 17:20:22.115182 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 2406
I20260923 17:20:22.115263 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:22.114925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43669 (local address 127.0.0.1:53408)
0923 17:20:22.115071 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:22.115146 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:43669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 17:20:22.116405 2710 ts_tablet_manager.cc:1918] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:20:22.116523 2710 log.cc:1196] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/5d5130de7054493ea0361a28ea994d2c
W20260923 17:20:22.116765 2190 catalog_manager.cc:4584] TS 981d09ef4d4a4546a03864b7e577731c (127.23.214.129:43669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d5130de7054493ea0361a28ea994d2c: Network error: Client connection negotiation failed: client connection to 127.23.214.129:43669: connect: Connection refused (error 111)
I20260923 17:20:22.116916 2710 ts_tablet_manager.cc:1939] T 5d5130de7054493ea0361a28ea994d2c P 4408553f3dc147649468378a4c57900f: Deleting consensus metadata
I20260923 17:20:22.118343 2189 catalog_manager.cc:4841] TS 4408553f3dc147649468378a4c57900f (127.23.214.131:37937): tablet 5d5130de7054493ea0361a28ea994d2c (table test-table [id=ef34c505c413458db9dab605a7949a15]) successfully deleted
I20260923 17:20:22.123360 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 2540
I20260923 17:20:22.123584 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:20:22.123294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:37443 (local address 127.0.0.1:45100)
0923 17:20:22.123442 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 17:20:22.123498 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.130:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 17:20:22.129256 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 2171
2026-09-23T17:20:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2534 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:20:22 dist-test-slave-9q7f krb5kdc[2716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:20:22 dist-test-slave-9q7f krb5kdc[2716](info): set up 2 sockets
Sep 23 17:20:22 dist-test-slave-9q7f krb5kdc[2716](info): commencing operation
krb5kdc: starting...
W20260923 17:20:23.233735 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:20:23 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:20:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/rangeradmin_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/rangeradmin_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/rangerlookup_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/rangerlookup_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
I20260923 17:20:23.409178 24410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskADzyP8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/postgres -l logfile start
2026-09-23 17:20:26.038 UTC [2746] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:20:26.038 UTC [2746] LOG: listening on IPv4 address "127.23.214.148", port 56209
2026-09-23 17:20:26.042 UTC [2746] LOG: listening on Unix socket "/tmp/.s.PGSQL.56209"
2026-09-23 17:20:26.064 UTC [2749] LOG: database system was shut down at 2026-09-23 17:20:24 UTC
2026-09-23 17:20:26.068 UTC [2746] LOG: database system is ready to accept connections
I20260923 17:20:27.028733 24410 mini_postgres.cc:96] Postgres bound to 56209
2026-09-23 17:20:27.035 UTC [2757] FATAL: database "slave" does not exist
127.23.214.148:56209 - accepting connections
I20260923 17:20:27.036332 24410 mini_ranger.cc:161] Starting Ranger...
I20260923 17:20:27.052570 24410 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:20:27.581722 24410 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:20:27.581825 24410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-admin
2026-09-23 17:20:27,869 [I] DB FLAVOR :POSTGRES
2026-09-23 17:20:27,870 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:20:27,870 [I] Checking connection..
2026-09-23 17:20:27,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:20:28,140 [I] Checking connection passed.
2026-09-23 17:20:28,141 [I] --------- Verifying version history table ---------
2026-09-23 17:20:28,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:20:28,409 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:20:28,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:20:28,678 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:20:28,678 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:20:28,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:20:28.907 UTC [2822] WARNING: there is no transaction in progress
2026-09-23 17:20:28,914 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:20:28,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:20:29,218 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:20:29,218 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:20:29,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:20:29,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:20:29,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9q7f.c.gcp-upstream.internal','N') ;"
2026-09-23 17:20:29,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:20:30,168 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:20:30,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:20:30,419 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:20:30,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:20:30,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:20:30,914 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:20:30,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:20:31.922 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:31.940 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:31.957 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:31.974 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:31.993 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.013 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.086 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.094 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.109 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.121 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.132 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.146 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.157 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.164 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.172 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32.179 UTC [2931] WARNING: there is no transaction in progress
2026-09-23 17:20:32,899 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:20:32,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:20:33,158 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:20:33,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:20:33,441 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:20:33,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:20:33,725 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:20:33,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:20:33,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:20:34,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9q7f.c.gcp-upstream.internal';"
2026-09-23 17:20:34,490 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:20:34,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:20:34,733 [I] DB_PATCHES have already been applied
I20260923 17:20:34.738775 24410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-admin:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskADzyP8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:20:34.738869 24410 mini_ranger.cc:193] Using host: 127.23.214.148
I20260923 17:20:34.742465 24410 mini_ranger.cc:227] Ranger admin URL: http://127.23.214.148:33901
Sep 23, 2026 5:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:20:35.864435 24410 mini_ranger.cc:160] Time spent starting Ranger: real 8.828s user 0.000s sys 0.010s
Sep 23, 2026 5:20:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:20:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 5:20:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:20:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:20:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:20:56.953069 24410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/rangerkms_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/rangerkms_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.214.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.214.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:20:57 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184057, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:20:57.044355 24410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:20:57.054948 24410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:20:57.999918 24410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:20:57.999997 24410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms
I20260923 17:20:58.000094 24410 mini_ranger_kms.cc:227] Using postgres at 127.23.214.148:56209
2026-09-23 17:20:58,182 [I] DB FLAVOR :POSTGRES
2026-09-23 17:20:58,183 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:20:58,183 [I] Checking connection
2026-09-23 17:20:58,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:20:58,415 [I] connection success
2026-09-23 17:20:58,415 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:20:58,415 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:20:58,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:20:58,662 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:20:58,662 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:20:58,662 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:20:58,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:20:58,925 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:20:59.453208 24410 mini_ranger_kms.cc:310] Created kms service
I20260923 17:20:59.673377 24410 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:20:59.718627 24410 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:20:59 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184057, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:00.109534 24410 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:21:00.109707 24410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms:/tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:21:00.109743 24410 mini_ranger_kms.cc:240] Using host: 127.23.214.148
I20260923 17:21:00.113642 24410 mini_ranger_kms.cc:276] Ranger KMS PID: 3112
I20260923 17:21:00.113796 24410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.214.148:52171
Sep 23, 2026 5:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 17:21:02 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184062, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.214.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:02.433851 24410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.390s user 0.015s sys 0.153s
I20260923 17:21:02.434033 24410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:21:02.434165 24410 mini_ranger_kms.cc:401] 127.23.214.148:52171/kms/v1/keys
Sep 23 17:21:02 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184062, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.214.148@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
Sep 23 17:21:03 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184062, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.214.148@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
Sep 23 17:21:05 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:21:05.561494 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:37709
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:37709
--ranger_config_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:52171/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:05.708678 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:05.709015 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:05.709057 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:05.713292 3164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:05.713371 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:05.713392 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:05.713408 3164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:05.713423 3164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:05.719379 3164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.214.148:52171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:37709
--ranger_config_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:05.720552 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:05.721625 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:05.731034 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:05.731050 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:05.731022 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:05.731128 3164 server_base.cc:1034] running on GCE node
I20260923 17:21:05.731946 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:05.733320 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:05.734498 3164 hybrid_clock.cc:648] HybridClock initialized: now 1790184065734462 us; error 47 us; skew 500 ppm
Sep 23 17:21:05 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:05.742621 3164 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:21:05.743754 3164 webserver.cc:460] Webserver started at http://127.23.214.190:34647/ using document root <none> and password file <none>
I20260923 17:21:05.744354 3164 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:05.744441 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:05.744700 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:05 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:05.793322 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "2fbb0462c15c45c99104665055d5263e"
format_stamp: "Formatted at 2026-09-23 17:21:05 on dist-test-slave-9q7f"
server_key: "1fa25f283cdac6f37a2aaccd305e2140"
server_key_iv: "446ea7f814ed0215ebfae30eee249faa"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:05.794037 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "2fbb0462c15c45c99104665055d5263e"
format_stamp: "Formatted at 2026-09-23 17:21:05 on dist-test-slave-9q7f"
server_key: "1fa25f283cdac6f37a2aaccd305e2140"
server_key_iv: "446ea7f814ed0215ebfae30eee249faa"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:05.798498 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:21:05.817757 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:05.819375 3164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:05.819499 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "2fbb0462c15c45c99104665055d5263e"
format_stamp: "Formatted at 2026-09-23 17:21:05 on dist-test-slave-9q7f"
server_key: "1fa25f283cdac6f37a2aaccd305e2140"
server_key_iv: "446ea7f814ed0215ebfae30eee249faa"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:05.819661 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:05.837253 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:05.840556 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:05.840770 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:05.848958 3164 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:37709
I20260923 17:21:05.849016 3231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:37709 every 8 connection(s)
I20260923 17:21:05.850049 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:05.850517 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 3164
I20260923 17:21:05.850637 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:21:05.850925 24410 external_mini_cluster.cc:1428] Setting key 3588750216f0ecd9500086e71a740b6a
I20260923 17:21:05.853267 3232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:05.859898 3232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e: Bootstrap starting.
I20260923 17:21:05.862655 3232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e: Neither blocks nor log segments found. Creating new log.
Sep 23 17:21:05 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:05.863467 3232 log.cc:824] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:05.865559 3232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e: No bootstrap required, opened a new log
I20260923 17:21:05.869388 3232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb0462c15c45c99104665055d5263e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37709 } }
I20260923 17:21:05.869663 3232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:05.869740 3232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbb0462c15c45c99104665055d5263e, State: Initialized, Role: FOLLOWER
I20260923 17:21:05.870152 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:05.852387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.23.214.190:37709)
0923 17:21:05.852880 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:05.852900 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:05.854289 (+ 1389us) server_negotiation.cc:407] Connection header received
0923 17:21:05.855261 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:05.855285 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:05.855716 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:05.856104 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:05.857014 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:05.858615 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:05.859836 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:05.860570 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:05.860727 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:05.863846 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:05.863856 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:05.863873 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:05.863904 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:05.866627 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:05.867155 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:05.867164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:05.867173 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:05.867337 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:05.867703 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:05.867707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:05.867709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:05.868273 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:05.868460 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:05.868799 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 17:21:05.869102 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":151,"threads_started":1}
I20260923 17:21:05.870378 3232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb0462c15c45c99104665055d5263e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37709 } }
I20260923 17:21:05.870550 3232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:05.870621 3232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:05.870718 3232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:05.871857 3232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb0462c15c45c99104665055d5263e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37709 } }
I20260923 17:21:05.872227 3232 leader_election.cc:302] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fbb0462c15c45c99104665055d5263e; no voters:
I20260923 17:21:05.872575 3232 leader_election.cc:288] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:05.872808 3237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:05.873118 3237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [term 1 LEADER]: Becoming Leader. State: Replica: 2fbb0462c15c45c99104665055d5263e, State: Running, Role: LEADER
I20260923 17:21:05.873418 3237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb0462c15c45c99104665055d5263e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37709 } }
I20260923 17:21:05.874008 3232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:05.874643 3237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fbb0462c15c45c99104665055d5263e. Latest consensus state: current_term: 1 leader_uuid: "2fbb0462c15c45c99104665055d5263e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb0462c15c45c99104665055d5263e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37709 } } }
I20260923 17:21:05.874763 3237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:05.875269 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fbb0462c15c45c99104665055d5263e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbb0462c15c45c99104665055d5263e" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37709 } } }
I20260923 17:21:05.875375 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:05.876571 3232 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:21:05.877928 3242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:05.881196 3242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:05.890755 3242 catalog_manager.cc:1347] Generated new cluster ID: e9e685a0bbad49f9a90856aed4c001c0
I20260923 17:21:05.890837 3242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:05.907630 3242 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:05.908943 3242 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:05.919262 3242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e: Generated new TSK 0
I20260923 17:21:05.920104 3242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:21:06.641683 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:52171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:37709
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:06.762742 3278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:06.763048 3278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:06.763132 3278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:06.767850 3278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:06.767943 3278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:06.768035 3278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:21:06.774003 3278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.214.148:52171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:06.775318 3278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:06.776484 3278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:06.786530 3286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:06.786798 3283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:06.787333 3284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:06.787442 3278 server_base.cc:1034] running on GCE node
I20260923 17:21:06.787918 3278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:06.788606 3278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:06.789757 3278 hybrid_clock.cc:648] HybridClock initialized: now 1790184066789721 us; error 47 us; skew 500 ppm
Sep 23 17:21:06 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184066, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:06.793193 3278 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:21:06.794375 3278 webserver.cc:460] Webserver started at http://127.23.214.129:38113/ using document root <none> and password file <none>
I20260923 17:21:06.795094 3278 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:06.795195 3278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:06.795475 3278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:06 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184066, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:06.812033 3278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "288c4862d4684da7b00ca9fc39706bba"
format_stamp: "Formatted at 2026-09-23 17:21:06 on dist-test-slave-9q7f"
server_key: "507003bd8f1ece68759cb090821d0966"
server_key_iv: "774ac35080d4dc13d56433fe321899cc"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:06.812769 3278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "288c4862d4684da7b00ca9fc39706bba"
format_stamp: "Formatted at 2026-09-23 17:21:06 on dist-test-slave-9q7f"
server_key: "507003bd8f1ece68759cb090821d0966"
server_key_iv: "774ac35080d4dc13d56433fe321899cc"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:06.819638 3278 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260923 17:21:06.836452 3295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:06.838201 3278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:21:06.838336 3278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "288c4862d4684da7b00ca9fc39706bba"
format_stamp: "Formatted at 2026-09-23 17:21:06 on dist-test-slave-9q7f"
server_key: "507003bd8f1ece68759cb090821d0966"
server_key_iv: "774ac35080d4dc13d56433fe321899cc"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:06.838471 3278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:06.897719 3278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:06.901655 3278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:06.901853 3278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:06.902627 3278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:06.903829 3278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:06.903961 3278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:06.904109 3278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:06.904196 3278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:06.919237 3278 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:43507
I20260923 17:21:06.920423 3410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:43507 every 8 connection(s)
I20260923 17:21:06.922065 3278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:06.925753 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 3278
I20260923 17:21:06.925901 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:06.926225 24410 external_mini_cluster.cc:1428] Setting key 7a5a2997a534e4425fb69abaa837234c
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
Sep 23 17:21:06 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184066, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:06.956980 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:06.934536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:60077 (local address 127.23.214.190:37709)
0923 17:21:06.938943 (+ 4407us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:06.938948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:06.940781 (+ 1833us) server_negotiation.cc:407] Connection header received
0923 17:21:06.940839 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:06.940842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:06.940898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:06.940996 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:06.942634 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:06.944760 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:06.946707 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:06.947094 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:06.947205 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:06.952126 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:06.952131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:06.952134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:06.952165 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:06.954231 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:06.954962 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:06.954967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:06.954969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:06.955068 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:06.955535 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:06.955539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:06.955540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:06.955782 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:06.955869 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:06.956637 (+ 768us) server_negotiation.cc:299] Negotiation successful
0923 17:21:06.956776 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4305,"thread_start_us":83,"threads_started":1}
I20260923 17:21:06.958894 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:06.935086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:37709 (local address 127.23.214.129:60077)
0923 17:21:06.939247 (+ 4161us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:06.939317 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:06.940427 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:06.941256 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:06.941266 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:06.941687 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:06.942437 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:06.942457 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:06.944931 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:06.944935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:06.946524 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:06.946537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:06.947402 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:06.947405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:06.947615 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:06.948343 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:06.948367 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:06.951875 (+ 3508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:06.954382 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:06.954389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:06.954409 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:06.954828 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:06.955188 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:06.955191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:06.955193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:06.955399 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:06.955938 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:06.955945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:06.956305 (+ 360us) client_negotiation.cc:769] Sending connection context
0923 17:21:06.957308 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0923 17:21:06.957708 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3915,"thread_start_us":156,"threads_started":1}
I20260923 17:21:06.960539 3411 heartbeater.cc:347] Connected to a master server at 127.23.214.190:37709
I20260923 17:21:06.960826 3411 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:06.961413 3411 heartbeater.cc:511] Master 127.23.214.190:37709 requested a full tablet report, sending...
I20260923 17:21:06.963200 3196 ts_manager.cc:194] Registered new tserver with Master: 288c4862d4684da7b00ca9fc39706bba (127.23.214.129:43507)
I20260923 17:21:06.964699 3196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:60077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:21:06.969419 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:52171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:37709
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:21:06 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184066, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:21:07.147816 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:07.148262 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:07.148356 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:07.154965 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:07.155058 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:07.155153 3417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:21:07.163575 3417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.214.148:52171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:07.165197 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:07.166699 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:07.179800 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:07.179829 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:07.180198 3417 server_base.cc:1034] running on GCE node
W20260923 17:21:07.180872 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:07.181445 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:07.182368 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:07.185858 3417 hybrid_clock.cc:648] HybridClock initialized: now 1790184067185778 us; error 93 us; skew 500 ppm
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:07.189836 3417 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:21:07.191109 3417 webserver.cc:460] Webserver started at http://127.23.214.130:46609/ using document root <none> and password file <none>
I20260923 17:21:07.191869 3417 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:07.191951 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:07.192204 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:07.212853 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "66b75b042e424bc28e3ba8edea34420e"
format_stamp: "Formatted at 2026-09-23 17:21:07 on dist-test-slave-9q7f"
server_key: "6662529df3f7c1e5d1a0f6b13dbc90a5"
server_key_iv: "a90e42f886bdfa59f2c9d3484f335eb0"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:07.213707 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "66b75b042e424bc28e3ba8edea34420e"
format_stamp: "Formatted at 2026-09-23 17:21:07 on dist-test-slave-9q7f"
server_key: "6662529df3f7c1e5d1a0f6b13dbc90a5"
server_key_iv: "a90e42f886bdfa59f2c9d3484f335eb0"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:07.222265 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260923 17:21:07.244107 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:07.245996 3417 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:07.246219 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "66b75b042e424bc28e3ba8edea34420e"
format_stamp: "Formatted at 2026-09-23 17:21:07 on dist-test-slave-9q7f"
server_key: "6662529df3f7c1e5d1a0f6b13dbc90a5"
server_key_iv: "a90e42f886bdfa59f2c9d3484f335eb0"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:07.246455 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:07.265733 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:07.269701 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:07.270002 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:07.270828 3417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:07.271907 3417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:07.271970 3417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:07.272024 3417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:07.272053 3417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:07.283246 3417 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:40281
I20260923 17:21:07.283423 3547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:40281 every 8 connection(s)
I20260923 17:21:07.284438 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:07.291107 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 3417
I20260923 17:21:07.291246 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:07.291546 24410 external_mini_cluster.cc:1428] Setting key 4c4878b7d9ddebcffb8adc9b1796ba8f
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:21:07.305249 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.286671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:50329 (local address 127.23.214.190:37709)
0923 17:21:07.286817 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.286824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.290496 (+ 3672us) server_negotiation.cc:407] Connection header received
0923 17:21:07.291730 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.291735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.291795 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.291852 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:07.293601 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.294369 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.296243 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.296654 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.296751 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.300381 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:07.300386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:07.300390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:07.300425 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:07.302577 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:07.303277 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:07.303284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:07.303287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:07.303395 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:07.303843 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:07.303847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:07.303848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:07.304080 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:07.304181 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.304892 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.305053 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:21:07.307240 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.287179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:37709 (local address 127.23.214.130:50329)
0923 17:21:07.290262 (+ 3083us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:07.290331 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:07.291441 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:07.292333 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:07.292342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:07.292728 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:07.293423 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.293443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.294536 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.294540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.296075 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.296090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.297108 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.297114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.297369 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.298151 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:07.298181 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:07.300136 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:07.302730 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:07.302740 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:07.302763 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:07.303149 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:07.303525 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:07.303528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:07.303530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:07.303745 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:07.304184 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:07.304190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:07.304588 (+ 398us) client_negotiation.cc:769] Sending connection context
0923 17:21:07.305684 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0923 17:21:07.306111 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2812,"thread_start_us":158,"threads_started":1}
I20260923 17:21:07.311054 3548 heartbeater.cc:347] Connected to a master server at 127.23.214.190:37709
I20260923 17:21:07.311407 3548 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:07.312243 3548 heartbeater.cc:511] Master 127.23.214.190:37709 requested a full tablet report, sending...
I20260923 17:21:07.313424 3196 ts_manager.cc:194] Registered new tserver with Master: 66b75b042e424bc28e3ba8edea34420e (127.23.214.130:40281)
I20260923 17:21:07.314083 3196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:50329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:21:07.342042 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:52171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:37709
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184066, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
W20260923 17:21:07.511816 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:07.512223 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:07.512295 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:07.518544 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:07.518636 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:07.518723 3553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:21:07.526650 3553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.214.148:52171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:07.528066 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:07.529439 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:07.544064 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:07.544446 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:07.544463 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:07.545419 3553 server_base.cc:1034] running on GCE node
I20260923 17:21:07.545926 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:07.546726 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:07.547916 3553 hybrid_clock.cc:648] HybridClock initialized: now 1790184067547850 us; error 80 us; skew 500 ppm
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:07.551529 3553 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:21:07.552696 3553 webserver.cc:460] Webserver started at http://127.23.214.131:40487/ using document root <none> and password file <none>
I20260923 17:21:07.553426 3553 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:07.553509 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:07.553745 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:07.586134 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "db91d8b62c144060910e10fe35c95d2b"
format_stamp: "Formatted at 2026-09-23 17:21:07 on dist-test-slave-9q7f"
server_key: "7546c6dbeae39cd7b7cef5ffcde0ffb1"
server_key_iv: "e77f31e58204c20b652ad0caee46e4d7"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:07.586918 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "db91d8b62c144060910e10fe35c95d2b"
format_stamp: "Formatted at 2026-09-23 17:21:07 on dist-test-slave-9q7f"
server_key: "7546c6dbeae39cd7b7cef5ffcde0ffb1"
server_key_iv: "e77f31e58204c20b652ad0caee46e4d7"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:07.598265 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.002s
I20260923 17:21:07.642462 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:07.644522 3553 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.004s sys 0.000s
I20260923 17:21:07.644690 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "db91d8b62c144060910e10fe35c95d2b"
format_stamp: "Formatted at 2026-09-23 17:21:07 on dist-test-slave-9q7f"
server_key: "7546c6dbeae39cd7b7cef5ffcde0ffb1"
server_key_iv: "e77f31e58204c20b652ad0caee46e4d7"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:07.644845 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:07.663771 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:07.667694 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:07.667904 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:07.668625 3553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:07.669562 3553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:07.669647 3553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:07.669715 3553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:07.669747 3553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:07.680992 3553 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:36671
I20260923 17:21:07.681012 3682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:36671 every 8 connection(s)
I20260923 17:21:07.682113 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:07.682760 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 3553
I20260923 17:21:07.682930 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:07.683223 24410 external_mini_cluster.cc:1428] Setting key 5f6cecf1c0c9b6fd9de4dfd5e7cad59b
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:07.699805 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.688876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:51261 (local address 127.23.214.190:37709)
0923 17:21:07.688989 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.688993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.689008 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:21:07.689518 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.689521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.689573 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.689649 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:07.690813 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.691344 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.692483 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.692708 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.692820 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.695344 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:07.695347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:07.695349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:07.695374 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:07.697220 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:07.697844 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:07.697848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:07.697850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:07.697977 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:07.698406 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:07.698410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:07.698411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:07.698646 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:07.698724 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.699504 (+ 780us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.699630 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:21:07.701774 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.687814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:37709 (local address 127.23.214.131:51261)
0923 17:21:07.688317 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:07.688375 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:07.689314 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:07.689821 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:07.689828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:07.690182 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:07.690672 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.690686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.691480 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.691483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.692358 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.692368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.692859 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.692863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.693045 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.693714 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:07.693732 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:07.695156 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:07.697345 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:07.697351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:07.697370 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:07.697728 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:07.698101 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:07.698104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:07.698106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:07.698295 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:07.698812 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:07.698820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:07.699199 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 17:21:07.700227 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0923 17:21:07.700605 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260923 17:21:07.703540 3683 heartbeater.cc:347] Connected to a master server at 127.23.214.190:37709
I20260923 17:21:07.703794 3683 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:07.704391 3683 heartbeater.cc:511] Master 127.23.214.190:37709 requested a full tablet report, sending...
I20260923 17:21:07.705612 3196 ts_manager.cc:194] Registered new tserver with Master: db91d8b62c144060910e10fe35c95d2b (127.23.214.131:36671)
I20260923 17:21:07.706264 3196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:51261
I20260923 17:21:07.712507 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:21:07.722893 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.714705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.23.214.190:37709)
0923 17:21:07.714828 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.714832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.715137 (+ 305us) server_negotiation.cc:407] Connection header received
0923 17:21:07.715289 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.715292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.715338 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.715432 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:07.716228 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.716757 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.717594 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.717838 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.717994 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.719511 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:07.719514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:07.719517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:07.719546 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:07.721033 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:07.721469 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:07.721474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:07.721476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:07.721559 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:07.721858 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:07.721862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:07.721863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:07.722117 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:07.722223 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.722591 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.722704 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:21:07.726223 3196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:07.729800 3196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:07.752758 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.740867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:36671 (local address 127.0.0.1:42900)
0923 17:21:07.743523 (+ 2656us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:07.743578 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:07.743754 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:07.747755 (+ 4001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:07.747758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:07.747777 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:07.747871 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.747877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.749866 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.749889 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.751055 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.751065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.752202 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.752206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.752319 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.752367 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 17:21:07.752467 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:21:07.752590 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2420,"thread_start_us":103,"threads_started":1}
I20260923 17:21:07.753686 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.740988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.23.214.131:36671)
0923 17:21:07.745288 (+ 4300us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.745301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.745325 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:21:07.745416 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.745421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.745637 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.748034 (+ 2397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:07.748193 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.749344 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.751229 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.752070 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.753191 (+ 1121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.753336 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.753387 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.753496 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4190,"thread_start_us":75,"threads_started":1}
I20260923 17:21:07.758958 3598 tablet_service.cc:1467] Processing CreateTablet for tablet 82da120c0f6942998d595884848dbbf5 (DEFAULT_TABLE table=test-table [id=20d64abe562c40b29af45e20712f5bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:07.760483 3598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82da120c0f6942998d595884848dbbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:07.768751 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.742033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40281 (local address 127.0.0.1:57876)
0923 17:21:07.744235 (+ 2202us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:07.744247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:07.744319 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:07.745833 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:07.745839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:07.745865 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:07.746061 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.746070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.749493 (+ 3423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.749497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.753518 (+ 4021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.753528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.768231 (+ 14703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.768235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.768345 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.768369 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:21:07.768464 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:21:07.768566 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"mutex_wait_us":2541,"thread_start_us":92,"threads_started":1}
I20260923 17:21:07.769604 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.742227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.23.214.130:40281)
0923 17:21:07.744623 (+ 2396us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.744637 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.744664 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:21:07.744747 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.744753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.744982 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.746275 (+ 1293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:07.746458 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.747603 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.766107 (+ 18504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.768074 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.768239 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.768400 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.769332 (+ 932us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.769466 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2234,"thread_start_us":151,"threads_started":1}
I20260923 17:21:07.773121 3464 tablet_service.cc:1467] Processing CreateTablet for tablet 82da120c0f6942998d595884848dbbf5 (DEFAULT_TABLE table=test-table [id=20d64abe562c40b29af45e20712f5bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:07.775569 3464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82da120c0f6942998d595884848dbbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:07.776099 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.741699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43507 (local address 127.0.0.1:57126)
0923 17:21:07.743982 (+ 2283us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:07.743994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:07.744077 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:07.748116 (+ 4039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:07.748119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:07.748134 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:07.748237 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.748243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.749686 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.749689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.752239 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.752247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.775622 (+ 23375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.775627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.775753 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.775786 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:21:07.775844 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 17:21:07.775946 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2226,"thread_start_us":86,"threads_started":1}
I20260923 17:21:07.776841 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.742031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.23.214.129:43507)
0923 17:21:07.742960 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.742970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.747630 (+ 4660us) server_negotiation.cc:407] Connection header received
0923 17:21:07.747721 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.747730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.747973 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.748379 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:07.748514 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.749573 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.774540 (+ 24967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.775460 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.776404 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.776555 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.776612 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.776707 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":549,"threads_started":1}
I20260923 17:21:07.778283 3702 tablet_bootstrap.cc:492] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Bootstrap starting.
I20260923 17:21:07.780229 3344 tablet_service.cc:1467] Processing CreateTablet for tablet 82da120c0f6942998d595884848dbbf5 (DEFAULT_TABLE table=test-table [id=20d64abe562c40b29af45e20712f5bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:07.781483 3344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82da120c0f6942998d595884848dbbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:07.781498 3702 tablet_bootstrap.cc:654] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:07.782516 3702 log.cc:824] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:07.784580 3705 tablet_bootstrap.cc:492] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e: Bootstrap starting.
I20260923 17:21:07.787031 3705 tablet_bootstrap.cc:654] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:07.787325 3702 tablet_bootstrap.cc:492] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: No bootstrap required, opened a new log
I20260923 17:21:07.787570 3702 ts_tablet_manager.cc:1397] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260923 17:21:07.788496 3705 log.cc:824] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:07.790582 3705 tablet_bootstrap.cc:492] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e: No bootstrap required, opened a new log
I20260923 17:21:07.790774 3705 ts_tablet_manager.cc:1397] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260923 17:21:07.791733 3702 raft_consensus.cc:348] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.792008 3702 raft_consensus.cc:374] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:07.792055 3702 raft_consensus.cc:729] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db91d8b62c144060910e10fe35c95d2b, State: Initialized, Role: FOLLOWER
I20260923 17:21:07.792523 3702 consensus_queue.cc:260] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.794003 3702 ts_tablet_manager.cc:1428] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260923 17:21:07.794353 3683 heartbeater.cc:503] Master 127.23.214.190:37709 was elected leader, sending a full tablet report...
I20260923 17:21:07.797260 3705 raft_consensus.cc:348] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.798120 3705 raft_consensus.cc:374] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:07.798590 3705 raft_consensus.cc:729] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b75b042e424bc28e3ba8edea34420e, State: Initialized, Role: FOLLOWER
I20260923 17:21:07.799386 3705 consensus_queue.cc:260] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.799795 3706 tablet_bootstrap.cc:492] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba: Bootstrap starting.
I20260923 17:21:07.800329 3548 heartbeater.cc:503] Master 127.23.214.190:37709 was elected leader, sending a full tablet report...
I20260923 17:21:07.800798 3705 ts_tablet_manager.cc:1428] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20260923 17:21:07.812340 3706 tablet_bootstrap.cc:654] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:07.813550 3706 log.cc:824] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:07.816315 3706 tablet_bootstrap.cc:492] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba: No bootstrap required, opened a new log
I20260923 17:21:07.816627 3706 ts_tablet_manager.cc:1397] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
Sep 23 17:21:07 dist-test-slave-9q7f krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184066, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:07.821796 3706 raft_consensus.cc:348] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.822228 3706 raft_consensus.cc:374] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:07.822330 3706 raft_consensus.cc:729] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288c4862d4684da7b00ca9fc39706bba, State: Initialized, Role: FOLLOWER
I20260923 17:21:07.823030 3706 consensus_queue.cc:260] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.825330 3706 ts_tablet_manager.cc:1428] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260923 17:21:07.825800 3411 heartbeater.cc:503] Master 127.23.214.190:37709 was elected leader, sending a full tablet report...
I20260923 17:21:07.837512 3708 raft_consensus.cc:482] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:07.837747 3708 raft_consensus.cc:504] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.839145 3708 leader_election.cc:288] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288c4862d4684da7b00ca9fc39706bba (127.23.214.129:43507), 66b75b042e424bc28e3ba8edea34420e (127.23.214.130:40281)
I20260923 17:21:07.845307 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.839517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43507 (local address 127.23.214.131:56369)
0923 17:21:07.840166 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:07.840179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:07.840293 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:07.841038 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:07.841041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:07.841061 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:07.841171 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.841177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.842251 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.842254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.843485 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.843496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.844887 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.844891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.845012 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.845035 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:21:07.845099 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 17:21:07.845172 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576}
I20260923 17:21:07.845901 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.840601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:56369 (local address 127.23.214.129:43507)
0923 17:21:07.840726 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.840730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.840745 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:21:07.840797 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.840800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.840846 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.840924 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:07.841281 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.842119 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.844053 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.844657 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.844773 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.844825 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.845678 (+ 853us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.845753 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:21:07.846995 3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da120c0f6942998d595884848dbbf5" candidate_uuid: "db91d8b62c144060910e10fe35c95d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288c4862d4684da7b00ca9fc39706bba" is_pre_election: true
I20260923 17:21:07.847358 3358 raft_consensus.cc:2454] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db91d8b62c144060910e10fe35c95d2b in term 0.
I20260923 17:21:07.847937 3569 leader_election.cc:302] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288c4862d4684da7b00ca9fc39706bba, db91d8b62c144060910e10fe35c95d2b; no voters:
I20260923 17:21:07.848374 3708 raft_consensus.cc:2781] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:07.848460 3708 raft_consensus.cc:482] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:07.848505 3708 raft_consensus.cc:3037] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:07.849859 3708 raft_consensus.cc:504] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.850275 3708 leader_election.cc:288] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [CANDIDATE]: Term 1 election: Requested vote from peers 288c4862d4684da7b00ca9fc39706bba (127.23.214.129:43507), 66b75b042e424bc28e3ba8edea34420e (127.23.214.130:40281)
I20260923 17:21:07.850906 3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da120c0f6942998d595884848dbbf5" candidate_uuid: "db91d8b62c144060910e10fe35c95d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288c4862d4684da7b00ca9fc39706bba"
I20260923 17:21:07.851035 3358 raft_consensus.cc:3037] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:07.851581 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.839930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:35655 (local address 127.23.214.130:40281)
0923 17:21:07.840066 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.840071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.844187 (+ 4116us) server_negotiation.cc:407] Connection header received
0923 17:21:07.844421 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.844424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.844481 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.844535 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:07.845039 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.845746 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.850127 (+ 4381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.850762 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.850877 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.850925 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.851358 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.851459 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:21:07.851931 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.839744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40281 (local address 127.23.214.131:35655)
0923 17:21:07.844087 (+ 4343us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:07.844102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:07.844273 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:07.844687 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:07.844690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:07.844707 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:07.844843 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.844849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.845935 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.845939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.848964 (+ 3025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:07.848976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.851121 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.851125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:07.851225 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.851247 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:21:07.851703 (+ 456us) client_negotiation.cc:240] Negotiation successful
0923 17:21:07.851781 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4264,"thread_start_us":148,"threads_started":1}
I20260923 17:21:07.852404 3358 raft_consensus.cc:2454] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db91d8b62c144060910e10fe35c95d2b in term 1.
I20260923 17:21:07.853148 3569 leader_election.cc:302] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288c4862d4684da7b00ca9fc39706bba, db91d8b62c144060910e10fe35c95d2b; no voters:
I20260923 17:21:07.853708 3708 raft_consensus.cc:2781] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:07.853996 3708 raft_consensus.cc:686] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 1 LEADER]: Becoming Leader. State: Replica: db91d8b62c144060910e10fe35c95d2b, State: Running, Role: LEADER
I20260923 17:21:07.854453 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da120c0f6942998d595884848dbbf5" candidate_uuid: "db91d8b62c144060910e10fe35c95d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b75b042e424bc28e3ba8edea34420e" is_pre_election: true
I20260923 17:21:07.854775 3484 raft_consensus.cc:2454] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db91d8b62c144060910e10fe35c95d2b in term 0.
I20260923 17:21:07.854480 3502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da120c0f6942998d595884848dbbf5" candidate_uuid: "db91d8b62c144060910e10fe35c95d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b75b042e424bc28e3ba8edea34420e"
I20260923 17:21:07.855137 3502 raft_consensus.cc:3037] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:07.856462 3502 raft_consensus.cc:2454] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db91d8b62c144060910e10fe35c95d2b in term 1.
I20260923 17:21:07.857502 3708 consensus_queue.cc:237] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } }
I20260923 17:21:07.861024 3196 catalog_manager.cc:5495] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b reported cstate change: term changed from 0 to 1, leader changed from <none> to db91d8b62c144060910e10fe35c95d2b (127.23.214.131). New cstate: current_term: 1 leader_uuid: "db91d8b62c144060910e10fe35c95d2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db91d8b62c144060910e10fe35c95d2b" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36671 } health_report { overall_health: HEALTHY } } }
I20260923 17:21:07.888449 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:07.883244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.23.214.131:36671)
0923 17:21:07.883379 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:07.883384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:07.883398 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:21:07.883482 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:07.883485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:07.883551 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:07.883653 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:07.884013 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.884798 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.886169 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:07.886509 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:07.886619 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:07.887074 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:07.887703 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:07.887837 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:07.888227 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 17:21:07.888290 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:21:07.896063 3358 raft_consensus.cc:1261] T 82da120c0f6942998d595884848dbbf5 P 288c4862d4684da7b00ca9fc39706bba [term 1 FOLLOWER]: Refusing update from remote peer db91d8b62c144060910e10fe35c95d2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:07.896040 3502 raft_consensus.cc:1261] T 82da120c0f6942998d595884848dbbf5 P 66b75b042e424bc28e3ba8edea34420e [term 1 FOLLOWER]: Refusing update from remote peer db91d8b62c144060910e10fe35c95d2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:07.896981 3713 consensus_queue.cc:1035] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b75b042e424bc28e3ba8edea34420e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:07.897447 3713 consensus_queue.cc:1035] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "288c4862d4684da7b00ca9fc39706bba" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:07.909724 3717 mvcc.cc:205] Tried to move back new op lower bound from 7332593942093574144 to 7332593941947949056. Current Snapshot: MvccSnapshot[applied={T|T < 7332593942093574144}]
I20260923 17:21:07.924098 3718 mvcc.cc:205] Tried to move back new op lower bound from 7332593942093574144 to 7332593941947949056. Current Snapshot: MvccSnapshot[applied={T|T < 7332593942093574144}]
I20260923 17:21:07.933259 3196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:07.933463 3196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:07.936334 3196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fbb0462c15c45c99104665055d5263e: Sending DeleteTablet for 3 replicas of tablet 82da120c0f6942998d595884848dbbf5
I20260923 17:21:07.937541 3344 tablet_service.cc:1514] Processing DeleteTablet for tablet 82da120c0f6942998d595884848dbbf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:07 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:57126
I20260923 17:21:07.937668 3464 tablet_service.cc:1514] Processing DeleteTablet for tablet 82da120c0f6942998d595884848dbbf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:07 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:57876
I20260923 17:21:07.938988 3598 tablet_service.cc:1514] Processing DeleteTablet for tablet 82da120c0f6942998d595884848dbbf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:07 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:42900
I20260923 17:21:07.940472 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 3278
I20260923 17:21:07.944169 3719 mvcc.cc:205] Tried to move back new op lower bound from 7332593942093574144 to 7332593941947949056. Current Snapshot: MvccSnapshot[applied={T|T < 7332593942093574144}]
I20260923 17:21:07.949393 3727 tablet_replica.cc:330] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: stopping tablet replica
I20260923 17:21:07.949951 3727 raft_consensus.cc:2229] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:07.950491 3727 raft_consensus.cc:2258] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 17:21:07.958108 3180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:43507 (error 108)
I20260923 17:21:07.966089 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 3417
I20260923 17:21:07.972155 3727 ts_tablet_manager.cc:1905] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:07.997648 3727 ts_tablet_manager.cc:1918] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:07.997822 3727 log.cc:1196] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/82da120c0f6942998d595884848dbbf5
I20260923 17:21:07.998298 3727 ts_tablet_manager.cc:1939] T 82da120c0f6942998d595884848dbbf5 P db91d8b62c144060910e10fe35c95d2b: Deleting consensus metadata
I20260923 17:21:08.000005 3183 catalog_manager.cc:4841] TS db91d8b62c144060910e10fe35c95d2b (127.23.214.131:36671): tablet 82da120c0f6942998d595884848dbbf5 (table test-table [id=20d64abe562c40b29af45e20712f5bb3]) successfully deleted
I20260923 17:21:08.008785 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:08.008416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43507 (local address 127.0.0.1:57134)
0923 17:21:08.008613 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:08.008685 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260923 17:21:08.009162 3180 catalog_manager.cc:4584] TS 288c4862d4684da7b00ca9fc39706bba (127.23.214.129:43507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82da120c0f6942998d595884848dbbf5: Network error: Client connection negotiation failed: client connection to 127.23.214.129:43507: connect: Connection refused (error 111)
W20260923 17:21:08.010524 3697 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:21:08.009955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40281 (local address 127.0.0.1:57888)
0923 17:21:08.010118 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:08.010132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:08.010223 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:08.010437 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.130:40281: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:21:08.011284 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 3553
I20260923 17:21:08.022193 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 3164
I20260923 17:21:08.031672 24410 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:21:08.380327 24410 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:21:08.380456 24410 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:21:08.725769 24410 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:21:08.726 UTC [2746] LOG: received smart shutdown request
2026-09-23 17:21:08.729 UTC [2746] LOG: background worker "logical replication launcher" (PID 2755) exited with exit code 1
2026-09-23 17:21:08.730 UTC [2750] LOG: shutting down
2026-09-23 17:21:08.914 UTC [2746] LOG: database system is shut down
2026-09-23T17:21:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46906 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:09 dist-test-slave-9q7f krb5kdc[3738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:09 dist-test-slave-9q7f krb5kdc[3738](info): set up 2 sockets
Sep 23 17:21:09 dist-test-slave-9q7f krb5kdc[3738](info): commencing operation
krb5kdc: starting...
W20260923 17:21:10.148211 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:21:10 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/rangeradmin_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/rangeradmin_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/rangerlookup_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/rangerlookup_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
I20260923 17:21:10.329485 24410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskADzyP8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/postgres -l logfile start
2026-09-23 17:21:12.933 UTC [3768] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:21:12.933 UTC [3768] LOG: listening on IPv4 address "127.23.214.148", port 56965
2026-09-23 17:21:12.936 UTC [3768] LOG: listening on Unix socket "/tmp/.s.PGSQL.56965"
2026-09-23 17:21:12.958 UTC [3771] LOG: database system was shut down at 2026-09-23 17:21:11 UTC
2026-09-23 17:21:12.963 UTC [3768] LOG: database system is ready to accept connections
I20260923 17:21:13.948107 24410 mini_postgres.cc:96] Postgres bound to 56965
2026-09-23 17:21:13.954 UTC [3779] FATAL: database "slave" does not exist
127.23.214.148:56965 - accepting connections
I20260923 17:21:13.955269 24410 mini_ranger.cc:161] Starting Ranger...
I20260923 17:21:13.971046 24410 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:21:14.474112 24410 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:21:14.474206 24410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-admin
2026-09-23 17:21:14,761 [I] DB FLAVOR :POSTGRES
2026-09-23 17:21:14,761 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:21:14,762 [I] Checking connection..
2026-09-23 17:21:14,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:21:15,016 [I] Checking connection passed.
2026-09-23 17:21:15,016 [I] --------- Verifying version history table ---------
2026-09-23 17:21:15,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:21:15,260 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:21:15,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:21:15,515 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:21:15,516 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:21:15,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:21:15.751 UTC [3844] WARNING: there is no transaction in progress
2026-09-23 17:21:15,758 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:21:15,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:21:16,018 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:21:16,018 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:21:16,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:21:16,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:21:16,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9q7f.c.gcp-upstream.internal','N') ;"
2026-09-23 17:21:16,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:21:16,994 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:21:16,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:21:17,261 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:21:17,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:21:17,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:21:17,785 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:21:17,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:21:18.815 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.833 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.851 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.868 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.885 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.904 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.971 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.979 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:18.994 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19.005 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19.016 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19.035 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19.046 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19.053 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19.060 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19.067 UTC [3952] WARNING: there is no transaction in progress
2026-09-23 17:21:19,735 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:21:19,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:21:19,989 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:21:19,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:21:20,222 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:21:20,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:21:20,471 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:21:20,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:21:20,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:21:20,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9q7f.c.gcp-upstream.internal';"
2026-09-23 17:21:21,212 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:21:21,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:21:21,460 [I] DB_PATCHES have already been applied
I20260923 17:21:21.466230 24410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-admin:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskADzyP8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:21:21.466300 24410 mini_ranger.cc:193] Using host: 127.23.214.148
I20260923 17:21:21.469740 24410 mini_ranger.cc:227] Ranger admin URL: http://127.23.214.148:41841
Sep 23, 2026 5:21:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:21:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:21:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:21:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:21:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:21:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:21:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:21:22.585467 24410 mini_ranger.cc:160] Time spent starting Ranger: real 8.630s user 0.005s sys 0.005s
Sep 23, 2026 5:21:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:21:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 5:21:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:21:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:21:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:21:45.014743 24410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.214.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.214.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/rangerkms_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.214.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/rangerkms_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.214.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.214.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.214.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/HTTP_127.23.214.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:21:45 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184105, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:21:45.103025 24410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:21:45.112901 24410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:21:46.011777 24410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:21:46.011865 24410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms
I20260923 17:21:46.011888 24410 mini_ranger_kms.cc:227] Using postgres at 127.23.214.148:56965
2026-09-23 17:21:46,182 [I] DB FLAVOR :POSTGRES
2026-09-23 17:21:46,182 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:21:46,182 [I] Checking connection
2026-09-23 17:21:46,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:21:46,433 [I] connection success
2026-09-23 17:21:46,433 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:21:46,433 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:21:46,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:21:46,698 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:21:46,698 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:21:46,698 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:21:46,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.214.148:56965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:21:46,950 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:21:47.442510 24410 mini_ranger_kms.cc:310] Created kms service
I20260923 17:21:47.613489 24410 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:21:47.658041 24410 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:21:47 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184105, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:48.058118 24410 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:21:48.058270 24410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms:/tmp/dist-test-taskADzyP8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskADzyP8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:21:48.058291 24410 mini_ranger_kms.cc:240] Using host: 127.23.214.148
I20260923 17:21:48.062285 24410 mini_ranger_kms.cc:276] Ranger KMS PID: 4134
I20260923 17:21:48.062378 24410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.214.148:33825
Sep 23, 2026 5:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 17:21:50 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.214.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:50.357255 24410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.254s user 0.021s sys 0.133s
I20260923 17:21:50.357388 24410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:21:50.357470 24410 mini_ranger_kms.cc:401] 127.23.214.148:33825/kms/v1/keys
Sep 23 17:21:50 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.214.148@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
Sep 23 17:21:50 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.214.148@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
Sep 23 17:21:53 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:21:53.229331 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44811
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:44811
--ranger_config_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:53.340022 4185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:53.340345 4185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:53.340387 4185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:53.344614 4185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:53.344696 4185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:53.344717 4185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:53.344732 4185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:53.344748 4185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:53.350646 4185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.214.148:33825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:44811
--ranger_config_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:53.351820 4185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:53.352932 4185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:53.362413 4191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:53.362475 4185 server_base.cc:1034] running on GCE node
W20260923 17:21:53.362421 4190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:53.362414 4193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:53.363255 4185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:53.364501 4185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:53.365691 4185 hybrid_clock.cc:648] HybridClock initialized: now 1790184113365680 us; error 32 us; skew 500 ppm
Sep 23 17:21:53 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:53.368448 4185 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:21:53.369333 4185 webserver.cc:460] Webserver started at http://127.23.214.190:34877/ using document root <none> and password file <none>
I20260923 17:21:53.369956 4185 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:53.370049 4185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:53.370278 4185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:53 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:53.425776 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "5d6db39000464958a784845f93553523"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-9q7f"
server_key: "fc1ac110bf768acbc0078239fadd9375"
server_key_iv: "f5879bcb221c5db68ad4f64f107433fa"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:53.426610 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "5d6db39000464958a784845f93553523"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-9q7f"
server_key: "fc1ac110bf768acbc0078239fadd9375"
server_key_iv: "f5879bcb221c5db68ad4f64f107433fa"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:53.431617 4185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:21:53.449558 4200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:53.451368 4185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:21:53.451529 4185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "5d6db39000464958a784845f93553523"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-9q7f"
server_key: "fc1ac110bf768acbc0078239fadd9375"
server_key_iv: "f5879bcb221c5db68ad4f64f107433fa"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:53.451681 4185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:53.468292 4185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:53.471238 4185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:53.471401 4185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:53.478729 4185 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:44811
I20260923 17:21:53.478755 4253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:44811 every 8 connection(s)
I20260923 17:21:53.479671 4185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:53.482473 4254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:53.487233 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 4185
I20260923 17:21:53.487365 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:21:53.487665 24410 external_mini_cluster.cc:1428] Setting key d630eb3a955ca0e1ea2da813d0f7b95f
I20260923 17:21:53.488122 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Bootstrap starting.
I20260923 17:21:53.490638 4254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:53.491341 4254 log.cc:824] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:53.493090 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: No bootstrap required, opened a new log
Sep 23 17:21:53 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:53.496704 4254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:53.496932 4254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:53.497015 4254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6db39000464958a784845f93553523, State: Initialized, Role: FOLLOWER
I20260923 17:21:53.497488 4254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:53.497643 4254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:53.497728 4254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:53.497830 4254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:53.498772 4254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:53.499110 4254 leader_election.cc:302] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d6db39000464958a784845f93553523; no voters:
I20260923 17:21:53.499424 4254 leader_election.cc:288] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:21:53.499485 4259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:53.499737 4259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 LEADER]: Becoming Leader. State: Replica: 5d6db39000464958a784845f93553523, State: Running, Role: LEADER
I20260923 17:21:53.500025 4259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:53.500715 4254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:53.501856 4261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d6db39000464958a784845f93553523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } } }
I20260923 17:21:53.501866 4260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d6db39000464958a784845f93553523. Latest consensus state: current_term: 1 leader_uuid: "5d6db39000464958a784845f93553523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } } }
I20260923 17:21:53.502009 4261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:53.502020 4260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:53.502439 4268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:53.502506 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:53.489086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.23.214.190:44811)
0923 17:21:53.489574 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:53.489589 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:53.489623 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:21:53.490327 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:53.490346 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:53.490674 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:53.491058 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:53.491833 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.492777 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.493685 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:53.494212 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:53.494413 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:53.496900 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:53.496904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:53.496917 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:53.496963 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:53.498828 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.499329 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.499334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.499339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.499439 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:53.499876 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:53.499880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:53.499881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:53.500455 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:53.500660 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:53.500994 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 17:21:53.501278 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":155,"threads_started":1}
I20260923 17:21:53.505508 4268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:53.505961 4254 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:21:53.514750 4268 catalog_manager.cc:1347] Generated new cluster ID: 5ec865e8f7df496580a78dd62ba6c9b2
I20260923 17:21:53.514820 4268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:53.529793 4268 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:21:53.531075 4268 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:53.544064 4268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Generated new TSK 0
I20260923 17:21:53.544991 4268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:21:54.216776 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44811
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:54.363824 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:54.364185 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:54.364251 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:54.370150 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:54.370256 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:54.370357 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:21:54.377486 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.214.148:33825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:54.378937 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:54.380260 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:54.392045 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:54.392349 4299 server_base.cc:1034] running on GCE node
W20260923 17:21:54.392155 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:54.392043 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:54.392997 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:54.393669 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:54.394867 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790184114394846 us; error 51 us; skew 500 ppm
Sep 23 17:21:54 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:54.398444 4299 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:21:54.399535 4299 webserver.cc:460] Webserver started at http://127.23.214.129:32975/ using document root <none> and password file <none>
I20260923 17:21:54.400235 4299 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:54.400317 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:54.400557 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:54 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:54.419358 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "67fc17e4ed184857b4f6bd5955d4dfd4"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "b823a9a10ec4e2021ff8da1cb313c656"
server_key_iv: "b8ab16450f71bd4ce73d07d8510842cb"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:54.420222 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "67fc17e4ed184857b4f6bd5955d4dfd4"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "b823a9a10ec4e2021ff8da1cb313c656"
server_key_iv: "b8ab16450f71bd4ce73d07d8510842cb"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:54.431607 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260923 17:21:54.448276 4318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:54.449831 4299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:21:54.450004 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "67fc17e4ed184857b4f6bd5955d4dfd4"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "b823a9a10ec4e2021ff8da1cb313c656"
server_key_iv: "b8ab16450f71bd4ce73d07d8510842cb"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:54.450248 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:54.485551 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:54.489117 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:54.489353 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:54.490056 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:54.491071 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:54.491166 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:54.491273 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:54.491336 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:54.501374 4299 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:43749
I20260923 17:21:54.502463 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:54.503525 4431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:43749 every 8 connection(s)
I20260923 17:21:54.508563 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 4299
I20260923 17:21:54.508688 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:21:54.508984 24410 external_mini_cluster.cc:1428] Setting key 9209838b24eec82835d2f0369939ec7c
Sep 23 17:21:54 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:54.519397 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.504508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:54285 (local address 127.23.214.190:44811)
0923 17:21:54.504820 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.504829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.505633 (+ 804us) server_negotiation.cc:407] Connection header received
0923 17:21:54.506683 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.506687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.506746 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.506863 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:54.507962 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.508710 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.511096 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.511504 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.511599 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.514665 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:54.514672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:54.514676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:54.514708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:54.516711 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.517419 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.517426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.517431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.517570 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.518079 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.518084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.518089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.518354 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:54.518456 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.519062 (+ 606us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.519198 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260923 17:21:54.520475 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.504991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44811 (local address 127.23.214.129:54285)
0923 17:21:54.505467 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:54.505531 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:54.506456 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:54.507043 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:54.507050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:54.507323 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:54.507812 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.507825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.508871 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.508876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.510919 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.510932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.511820 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.511827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.512079 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.512855 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:54.512872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:54.514479 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:54.516859 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:54.516869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:54.516892 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:54.517293 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:54.517696 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:54.517700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:54.517702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:54.517953 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:54.518526 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:54.518532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:54.518794 (+ 262us) client_negotiation.cc:769] Sending connection context
0923 17:21:54.518998 (+ 204us) client_negotiation.cc:240] Negotiation successful
0923 17:21:54.519345 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":138,"threads_started":1}
I20260923 17:21:54.522092 4432 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44811
I20260923 17:21:54.522352 4432 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:54.522982 4432 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:54.524664 4217 ts_manager.cc:194] Registered new tserver with Master: 67fc17e4ed184857b4f6bd5955d4dfd4 (127.23.214.129:43749)
I20260923 17:21:54.525799 4217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:54285
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
Sep 23 17:21:54 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:21:54.549906 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44811
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:54.713421 4439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:54.713807 4439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:54.713912 4439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:54.720007 4439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:54.720150 4439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:54.720278 4439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:21:54.728152 4439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.214.148:33825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:54.729702 4439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:54.731344 4439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:54.743543 4448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:54.743544 4445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:54.744036 4439 server_base.cc:1034] running on GCE node
W20260923 17:21:54.744086 4446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:54.744668 4439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:54.745561 4439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:54.746779 4439 hybrid_clock.cc:648] HybridClock initialized: now 1790184114746746 us; error 45 us; skew 500 ppm
Sep 23 17:21:54 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:54.750624 4439 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:21:54.751673 4439 webserver.cc:460] Webserver started at http://127.23.214.130:40335/ using document root <none> and password file <none>
I20260923 17:21:54.752357 4439 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:54.752442 4439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:54.752662 4439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:54 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:54.778064 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "fc717344dd9a4d59a7d48c8a0a015e6d"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "ace20b5472a63f082eda7fecb92ba4bf"
server_key_iv: "483028ccbb99836dc9c66be5e8109915"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:54.778929 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "fc717344dd9a4d59a7d48c8a0a015e6d"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "ace20b5472a63f082eda7fecb92ba4bf"
server_key_iv: "483028ccbb99836dc9c66be5e8109915"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:54.785625 4439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 17:21:54.809127 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:54.811092 4439 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260923 17:21:54.811446 4439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "fc717344dd9a4d59a7d48c8a0a015e6d"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "ace20b5472a63f082eda7fecb92ba4bf"
server_key_iv: "483028ccbb99836dc9c66be5e8109915"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:54.811627 4439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:54.834640 4439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:54.838449 4439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:54.838681 4439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:54.839332 4439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:54.840252 4439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:54.840332 4439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:54.840402 4439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:54.840441 4439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:54.851140 4439 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:45023
I20260923 17:21:54.851541 4568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:45023 every 8 connection(s)
I20260923 17:21:54.852232 4439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:54.852643 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 4439
I20260923 17:21:54.852777 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:21:54.853175 24410 external_mini_cluster.cc:1428] Setting key 86c8217e588c152204f055c693018e95
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Sep 23 17:21:54 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:54.891695 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.855639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:39189 (local address 127.23.214.190:44811)
0923 17:21:54.855815 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:54.855823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:54.862510 (+ 6687us) server_negotiation.cc:407] Connection header received
0923 17:21:54.863996 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:54.864000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:54.864065 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:54.864126 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:54.866579 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.867427 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.869495 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.869912 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.870013 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.886872 (+ 16859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:54.886877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:54.886880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:54.886915 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:54.888761 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.889503 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.889507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.889509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.889600 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:54.890120 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:54.890125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:54.890127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:54.890395 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:54.890489 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:54.891317 (+ 828us) server_negotiation.cc:299] Negotiation successful
0923 17:21:54.891465 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:21:54.894157 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:54.856177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44811 (local address 127.23.214.130:39189)
0923 17:21:54.862310 (+ 6133us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:54.862384 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:54.863648 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:54.864608 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:54.864619 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:54.865032 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:54.866326 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.866353 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.867599 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.867603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.869331 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:54.869343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:54.880793 (+ 11450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:54.880799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:54.881172 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:54.883150 (+ 1978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:54.883177 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:54.886605 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:54.888941 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:54.888948 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:54.888969 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:54.889361 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:54.889720 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:54.889723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:54.889737 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:54.889991 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:54.890563 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:54.890570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:54.890949 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 17:21:54.892568 (+ 1619us) client_negotiation.cc:240] Negotiation successful
0923 17:21:54.893005 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5847,"thread_start_us":157,"threads_started":1}
I20260923 17:21:54.895406 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44811
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:21:54.898576 4569 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44811
I20260923 17:21:54.898859 4569 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:54.899530 4569 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:54.900790 4217 ts_manager.cc:194] Registered new tserver with Master: fc717344dd9a4d59a7d48c8a0a015e6d (127.23.214.130:45023)
I20260923 17:21:54.901588 4217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:39189
W20260923 17:21:55.092129 4574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:55.092535 4574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:55.092612 4574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:55.101521 4574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:55.101655 4574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:55.101760 4574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:21:55.110849 4574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.214.148:33825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:55.112293 4574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:55.113680 4574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184114, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
W20260923 17:21:55.127622 4580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.127635 4582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.130080 4579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:55.130477 4574 server_base.cc:1034] running on GCE node
I20260923 17:21:55.131006 4574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:55.131803 4574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:55.133031 4574 hybrid_clock.cc:648] HybridClock initialized: now 1790184115132962 us; error 85 us; skew 500 ppm
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:55.136826 4574 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:21:55.138065 4574 webserver.cc:460] Webserver started at http://127.23.214.131:40341/ using document root <none> and password file <none>
I20260923 17:21:55.138849 4574 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:55.139005 4574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:55.139305 4574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:55.161108 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "ef499ca2d8904f99abe4aed83a6913de"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-9q7f"
server_key: "1c83fcac4638f42777c386b8633f8a45"
server_key_iv: "4f4cd6c867b497e1ad1dbaa3b5b3feae"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:55.174767 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "ef499ca2d8904f99abe4aed83a6913de"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-9q7f"
server_key: "1c83fcac4638f42777c386b8633f8a45"
server_key_iv: "4f4cd6c867b497e1ad1dbaa3b5b3feae"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:55.180732 4574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:21:55.206890 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.208797 4574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:21:55.208935 4574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "ef499ca2d8904f99abe4aed83a6913de"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-9q7f"
server_key: "1c83fcac4638f42777c386b8633f8a45"
server_key_iv: "4f4cd6c867b497e1ad1dbaa3b5b3feae"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:55.209082 4574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:55.246546 4574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:55.250422 4574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:55.250681 4574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:55.251405 4574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:55.252511 4574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:55.252621 4574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.252718 4574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:55.252782 4574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.268005 4574 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:40165
I20260923 17:21:55.269366 4574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:55.271296 4702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:40165 every 8 connection(s)
I20260923 17:21:55.274115 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 4574
I20260923 17:21:55.274235 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:21:55.274547 24410 external_mini_cluster.cc:1428] Setting key 36a9d6866c12de0d5de9ac924915a06f
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:55.289793 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:55.273214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:41327 (local address 127.23.214.190:44811)
0923 17:21:55.273375 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:55.273381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:55.275523 (+ 2142us) server_negotiation.cc:407] Connection header received
0923 17:21:55.276823 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:55.276827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:55.276880 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:55.276931 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:55.278611 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.279484 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.281298 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.281711 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.281790 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:55.285277 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:55.285281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:55.285284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:55.285313 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:55.287245 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:55.287913 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:55.287917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:55.287919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:55.288020 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:55.288462 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:55.288465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:55.288467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:55.288690 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:55.288770 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:55.289483 (+ 713us) server_negotiation.cc:299] Negotiation successful
0923 17:21:55.289624 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:21:55.291545 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:55.273753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44811 (local address 127.23.214.131:41327)
0923 17:21:55.275365 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:55.275421 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:55.276516 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:55.277358 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:55.277367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:55.277723 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:55.278435 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:55.278454 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.279651 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.279658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:55.281154 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:55.281166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.282044 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.282048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:55.282255 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:55.283207 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:55.283235 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:55.285066 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:55.287386 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:55.287393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:55.287412 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:55.287790 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:55.288141 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:55.288144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:55.288146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:55.288350 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:55.288835 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:55.288842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:55.289196 (+ 354us) client_negotiation.cc:769] Sending connection context
0923 17:21:55.290017 (+ 821us) client_negotiation.cc:240] Negotiation successful
0923 17:21:55.290414 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"thread_start_us":158,"threads_started":1}
I20260923 17:21:55.293025 4703 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44811
I20260923 17:21:55.293279 4703 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:55.293915 4703 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:55.295110 4217 ts_manager.cc:194] Registered new tserver with Master: ef499ca2d8904f99abe4aed83a6913de (127.23.214.131:40165)
I20260923 17:21:55.295852 4217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:41327
I20260923 17:21:55.300211 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:21:55.301110 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 4299
I20260923 17:21:55.310492 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 4439
I20260923 17:21:55.319758 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 4574
I20260923 17:21:55.326670 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 4185
I20260923 17:21:55.335170 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44811
--webserver_interface=127.23.214.190
--webserver_port=34877
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:44811
--ranger_config_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:55.451733 4706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:55.451987 4706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:55.452024 4706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:55.455998 4706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:55.456060 4706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:55.456080 4706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:55.456096 4706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:55.456110 4706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:55.461465 4706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:44811
--ranger_config_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=34877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:55.462486 4706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:55.463441 4706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:55.472591 4714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:55.472637 4706 server_base.cc:1034] running on GCE node
W20260923 17:21:55.472666 4712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.472591 4711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:55.473390 4706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:55.474570 4706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:55.475782 4706 hybrid_clock.cc:648] HybridClock initialized: now 1790184115475750 us; error 46 us; skew 500 ppm
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:55.478478 4706 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:21:55.479342 4706 webserver.cc:460] Webserver started at http://127.23.214.190:34877/ using document root <none> and password file <none>
I20260923 17:21:55.479830 4706 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:55.479892 4706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:55.538339 4706 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:21:55.540553 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.541860 4706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:55.542062 4706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "5d6db39000464958a784845f93553523"
format_stamp: "Formatted at 2026-09-23 17:21:53 on dist-test-slave-9q7f"
server_key: "fc1ac110bf768acbc0078239fadd9375"
server_key_iv: "f5879bcb221c5db68ad4f64f107433fa"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:55.542622 4706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:55.563125 4706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:55.566232 4706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:55.566442 4706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:55.573690 4706 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:44811
I20260923 17:21:55.573724 4774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:44811 every 8 connection(s)
I20260923 17:21:55.574784 4706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:55.579138 4775 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:21:55.580751 4775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Bootstrap starting.
I20260923 17:21:55.583834 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 4706
I20260923 17:21:55.584313 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:43749
--local_ip_for_outbound_sockets=127.23.214.129
--tserver_master_addrs=127.23.214.190:44811
--webserver_port=32975
--webserver_interface=127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:21:55.597791 4775 log.cc:824] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:55.604434 4775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:21:55.604875 4775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Bootstrap complete.
I20260923 17:21:55.609119 4775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:55.609535 4775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6db39000464958a784845f93553523, State: Initialized, Role: FOLLOWER
I20260923 17:21:55.610157 4775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:55.610334 4775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:21:55.610419 4775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:21:55.610558 4775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:21:55.612437 4775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:55.612772 4775 leader_election.cc:302] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d6db39000464958a784845f93553523; no voters:
I20260923 17:21:55.613082 4775 leader_election.cc:288] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:21:55.613250 4779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:21:55.613588 4779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [term 2 LEADER]: Becoming Leader. State: Replica: 5d6db39000464958a784845f93553523, State: Running, Role: LEADER
I20260923 17:21:55.614105 4779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } }
I20260923 17:21:55.614365 4775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:21:55.615669 4780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5d6db39000464958a784845f93553523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } } }
I20260923 17:21:55.615708 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d6db39000464958a784845f93553523. Latest consensus state: current_term: 2 leader_uuid: "5d6db39000464958a784845f93553523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6db39000464958a784845f93553523" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44811 } } }
I20260923 17:21:55.615782 4780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:55.615805 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523 [sys.catalog]: This master's current role is: LEADER
I20260923 17:21:55.616650 4775 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:21:55.617324 4789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:21:55.623775 4789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:21:55.624646 4789 catalog_manager.cc:1259] Loaded cluster ID: 5ec865e8f7df496580a78dd62ba6c9b2
I20260923 17:21:55.624783 4789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:21:55.627663 4789 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:21:55.628736 4789 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Loaded TSK: 0
I20260923 17:21:55.630333 4789 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:21:55.727551 4777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:55.728035 4777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:55.728118 4777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:55.734202 4777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:55.734315 4777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:55.734403 4777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:21:55.742115 4777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:43749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=32975
--tserver_master_addrs=127.23.214.190:44811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:55.743566 4777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:55.744920 4777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:55.757536 4813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.758034 4812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:55.758149 4815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:55.758177 4777 server_base.cc:1034] running on GCE node
I20260923 17:21:55.758709 4777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:55.759508 4777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:55.760723 4777 hybrid_clock.cc:648] HybridClock initialized: now 1790184115760687 us; error 50 us; skew 500 ppm
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:55.764463 4777 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:21:55.765669 4777 webserver.cc:460] Webserver started at http://127.23.214.129:32975/ using document root <none> and password file <none>
I20260923 17:21:55.766486 4777 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:55.766619 4777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:55.803325 4777 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260923 17:21:55.807379 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.808993 4777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:55.809120 4777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "67fc17e4ed184857b4f6bd5955d4dfd4"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "b823a9a10ec4e2021ff8da1cb313c656"
server_key_iv: "b8ab16450f71bd4ce73d07d8510842cb"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:55.809847 4777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:55.838725 4777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:55.842347 4777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:55.842584 4777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:55.843263 4777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:55.844086 4777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:55.844165 4777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.844230 4777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:55.844280 4777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:55.856031 4777 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:43749
I20260923 17:21:55.857450 4777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:21:55.859081 4938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:43749 every 8 connection(s)
I20260923 17:21:55.863051 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 4777
I20260923 17:21:55.863555 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:45023
--local_ip_for_outbound_sockets=127.23.214.130
--tserver_master_addrs=127.23.214.190:44811
--webserver_port=40335
--webserver_interface=127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:21:55 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184115, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:55.882961 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:55.863118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:46157 (local address 127.23.214.190:44811)
0923 17:21:55.865961 (+ 2843us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:55.865984 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:55.866027 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:21:55.866323 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:55.866336 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:55.866725 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:55.867069 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:55.868438 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.870138 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.871883 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.872619 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.872782 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:55.875555 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:55.875568 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:55.875582 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:55.875609 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:55.878994 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:55.879697 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:55.879703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:55.879710 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:55.879854 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:55.880279 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:55.880283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:55.880285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:55.880809 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:55.880967 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:55.881535 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 17:21:55.881850 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2615,"thread_start_us":127,"threads_started":1}
I20260923 17:21:55.884428 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:55.861589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44811 (local address 127.23.214.129:46157)
0923 17:21:55.864192 (+ 2603us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:55.864249 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:55.865426 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:55.867196 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:55.867206 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:55.867592 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:55.868248 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:55.868271 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.870318 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.870322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:55.871736 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:55.871747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:55.873007 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:55.873014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:55.873188 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:55.873850 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:55.873868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:55.875345 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:55.879159 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:55.879165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:55.879182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:55.879534 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:55.879968 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:55.879971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:55.879973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:55.880164 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:55.880951 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:55.880958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:55.881214 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 17:21:55.881403 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 17:21:55.883295 (+ 1892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2348,"thread_start_us":177,"threads_started":1}
I20260923 17:21:55.887944 4939 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44811
I20260923 17:21:55.888171 4939 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:55.888772 4939 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
W20260923 17:21:55.891072 4939 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:44811 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:21:55.891186 4939 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:55.891350 4939 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:55.892231 4939 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:55.892385 4939 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
W20260923 17:21:55.893121 4939 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:21:56.035935 4943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:56.036367 4943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:56.036515 4943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:56.043080 4943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:56.043174 4943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:56.043285 4943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:21:56.051720 4943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:45023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=40335
--tserver_master_addrs=127.23.214.190:44811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:56.053472 4943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:56.054924 4943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:56.067795 4951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.068336 4948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.068257 4949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:56.068509 4943 server_base.cc:1034] running on GCE node
I20260923 17:21:56.069166 4943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:56.070014 4943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:56.073362 4943 hybrid_clock.cc:648] HybridClock initialized: now 1790184116073318 us; error 59 us; skew 500 ppm
Sep 23 17:21:56 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:56.077243 4943 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:21:56.078420 4943 webserver.cc:460] Webserver started at http://127.23.214.130:40335/ using document root <none> and password file <none>
I20260923 17:21:56.079028 4943 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:56.079137 4943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:21:56 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:56.111084 4943 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.000s
I20260923 17:21:56.123788 4960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.125730 4943 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260923 17:21:56.125913 4943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "fc717344dd9a4d59a7d48c8a0a015e6d"
format_stamp: "Formatted at 2026-09-23 17:21:54 on dist-test-slave-9q7f"
server_key: "ace20b5472a63f082eda7fecb92ba4bf"
server_key_iv: "483028ccbb99836dc9c66be5e8109915"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:56.126689 4943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:56.161263 4943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:56.164266 4943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:56.164469 4943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:56.165095 4943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:56.166010 4943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:56.166075 4943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.166178 4943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:56.166224 4943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.177084 4943 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:45023
I20260923 17:21:56.177242 5074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:45023 every 8 connection(s)
I20260923 17:21:56.178471 4943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:21:56.188589 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 4943
I20260923 17:21:56.189446 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.214.148:33825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:40165
--local_ip_for_outbound_sockets=127.23.214.131
--tserver_master_addrs=127.23.214.190:44811
--webserver_port=40341
--webserver_interface=127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:21:56 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:56.195717 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.181758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:41307 (local address 127.23.214.190:44811)
0923 17:21:56.181995 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.182000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.182925 (+ 925us) server_negotiation.cc:407] Connection header received
0923 17:21:56.184040 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.184043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.184094 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.184175 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.185624 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.186220 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.187866 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.188097 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.188177 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:56.191002 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.191006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.191009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:56.191039 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.193024 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:56.193717 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:56.193721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:56.193723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:56.193822 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:56.194302 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:56.194309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:56.194311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:56.194550 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.194625 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.195379 (+ 754us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.195531 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260923 17:21:56.197458 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.182267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44811 (local address 127.23.214.130:41307)
0923 17:21:56.182765 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:56.182817 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:56.183777 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:56.184381 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:56.184391 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:56.184777 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:56.185446 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:56.185471 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.186351 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.186354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:56.187726 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:56.187738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.188255 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.188259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:56.188452 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:56.189162 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:56.189186 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:56.190781 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:56.193169 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:56.193176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:56.193195 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:56.193568 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:56.193984 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:56.193988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:56.193990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:56.194178 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:56.194729 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:56.194736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:56.195087 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 17:21:56.195975 (+ 888us) client_negotiation.cc:240] Negotiation successful
0923 17:21:56.196341 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":149,"threads_started":1}
I20260923 17:21:56.198910 5075 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44811
I20260923 17:21:56.199174 5075 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:56.199770 5075 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
W20260923 17:21:56.201521 5075 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:44811 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:21:56.201656 5075 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:56.201843 5075 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:56.202718 5075 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:56.202919 5075 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
W20260923 17:21:56.203712 5075 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:21:56.340799 5078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:56.341301 5078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:56.341432 5078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:56.347666 5078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:56.347810 5078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:56.347940 5078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:21:56.354645 5078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.214.148:33825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:40165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=40341
--tserver_master_addrs=127.23.214.190:44811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:56.356299 5078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:56.357729 5078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:56.369052 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.368983 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:56.368983 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:56.369490 5078 server_base.cc:1034] running on GCE node
I20260923 17:21:56.370059 5078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:56.370783 5078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:56.371979 5078 hybrid_clock.cc:648] HybridClock initialized: now 1790184116371950 us; error 41 us; skew 500 ppm
Sep 23 17:21:56 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:56.375278 5078 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:21:56.376351 5078 webserver.cc:460] Webserver started at http://127.23.214.131:40341/ using document root <none> and password file <none>
I20260923 17:21:56.377065 5078 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:56.377238 5078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:21:56 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:56.395491 5078 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:21:56.398067 5101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.399459 5078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:21:56.399593 5078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "ef499ca2d8904f99abe4aed83a6913de"
format_stamp: "Formatted at 2026-09-23 17:21:55 on dist-test-slave-9q7f"
server_key: "1c83fcac4638f42777c386b8633f8a45"
server_key_iv: "4f4cd6c867b497e1ad1dbaa3b5b3feae"
server_key_version: "kuduclusterkey@0"
I20260923 17:21:56.400214 5078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:56.413612 5078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:56.416994 5078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:56.417204 5078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:56.417804 5078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:21:56.418674 5078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:21:56.418745 5078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.418819 5078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:21:56.418874 5078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:56.429575 5078 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:40165
I20260923 17:21:56.429644 5215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:40165 every 8 connection(s)
I20260923 17:21:56.430999 5078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:21:56.431703 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5078
I20260923 17:21:56.443002 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.432525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.23.214.190:44811)
0923 17:21:56.432673 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.432678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.432873 (+ 195us) server_negotiation.cc:407] Connection header received
0923 17:21:56.433040 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.433043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.433099 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.433193 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.434134 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.435003 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.436190 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.436541 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.436661 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:56.438565 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.438569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.438572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:56.438603 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.440429 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:56.440999 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:56.441003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:56.441005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:56.441105 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:56.441424 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:56.441428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:56.441429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:56.441678 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.441912 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.442682 (+ 770us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.442812 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 17:21:56 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:56.450929 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.435295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:45363 (local address 127.23.214.190:44811)
0923 17:21:56.435593 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:56.435601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:56.435615 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:21:56.437725 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:56.437728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:56.437779 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:56.437865 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:56.439492 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.440321 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.442084 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.442462 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.442560 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:56.446318 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:56.446322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:56.446325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:56.446352 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:56.448267 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:56.448923 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:56.448927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:56.448929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:56.449031 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:56.449470 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:56.449473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:56.449475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:56.449740 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:56.449848 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:56.450589 (+ 741us) server_negotiation.cc:299] Negotiation successful
0923 17:21:56.450742 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":122,"threads_started":1}
I20260923 17:21:56.451964 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:56.434275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44811 (local address 127.23.214.131:45363)
0923 17:21:56.434734 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:56.434785 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:56.437387 (+ 2602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:56.438183 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:56.438196 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:56.438545 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:56.439253 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:56.439275 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.440471 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.440475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:56.441944 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:56.441956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:56.442724 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:56.442728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:56.442938 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:56.443542 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:56.443563 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:56.446057 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:56.448397 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:56.448403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:56.448422 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:56.448790 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:56.449145 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:56.449149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:56.449151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:56.449341 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:56.449858 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:56.449865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:56.450268 (+ 403us) client_negotiation.cc:769] Sending connection context
0923 17:21:56.450482 (+ 214us) client_negotiation.cc:240] Negotiation successful
0923 17:21:56.450838 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260923 17:21:56.453358 5216 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44811
I20260923 17:21:56.453665 5216 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:56.454358 5216 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:56.456074 4739 ts_manager.cc:194] Registered new tserver with Master: ef499ca2d8904f99abe4aed83a6913de (127.23.214.131:40165)
I20260923 17:21:56.457764 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:45363
Sep 23 17:21:56 dist-test-slave-9q7f krb5kdc[3738](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:56.893322 4939 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:56.893625 4939 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:56.894451 4739 ts_manager.cc:194] Registered new tserver with Master: 67fc17e4ed184857b4f6bd5955d4dfd4 (127.23.214.129:43749)
I20260923 17:21:56.895273 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:46157
Sep 23 17:21:57 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:57.203984 5075 heartbeater.cc:464] Registering TS with master...
I20260923 17:21:57.204316 5075 heartbeater.cc:511] Master 127.23.214.190:44811 requested a full tablet report, sending...
I20260923 17:21:57.205228 4739 ts_manager.cc:194] Registered new tserver with Master: fc717344dd9a4d59a7d48c8a0a015e6d (127.23.214.130:45023)
I20260923 17:21:57.206082 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:41307
I20260923 17:21:57.206987 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:21:57.219091 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.209173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.23.214.190:44811)
0923 17:21:57.209393 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.209398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.209537 (+ 139us) server_negotiation.cc:407] Connection header received
0923 17:21:57.209679 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.209682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.209730 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.209824 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:57.210713 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.211527 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.212685 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.213037 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.213208 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.215065 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:57.215069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:57.215072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:57.215100 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:57.217150 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:57.217673 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:57.217676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:57.217678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:57.217769 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:57.218097 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:57.218100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:57.218102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:57.218326 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:57.218457 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.218766 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.218901 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260923 17:21:57.223407 4739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:21:57.227993 4739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:21:57.248381 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.240864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:40165 (local address 127.0.0.1:55330)
0923 17:21:57.242439 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.242453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:57.242549 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:57.243251 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:57.243257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:57.243281 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:57.243420 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.243426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.244898 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.244902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.246417 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.246428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.247826 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.247829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.247948 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.247986 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:21:57.248106 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 17:21:57.248221 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"thread_start_us":92,"threads_started":1}
I20260923 17:21:57.249037 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.241276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.23.214.131:40165)
0923 17:21:57.242773 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.242780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.242802 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:21:57.242874 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.242879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.243087 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.243235 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:57.243559 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.244763 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.246551 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.247506 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.247631 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.247763 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.248765 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.248903 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":97,"threads_started":1}
I20260923 17:21:57.249650 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.239751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43749 (local address 127.0.0.1:59776)
0923 17:21:57.241903 (+ 2152us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.241953 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:57.242087 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:57.244050 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:57.244053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:57.244069 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:57.244159 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.244164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.246306 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.246309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.247414 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.247425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.249286 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.249289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.249373 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.249393 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:21:57.249459 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 17:21:57.249522 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1954,"thread_start_us":96,"threads_started":1}
I20260923 17:21:57.250253 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.239879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.23.214.129:43749)
0923 17:21:57.243512 (+ 3633us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.243518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.243550 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:21:57.243618 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.243623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.243786 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.243919 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:57.244995 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.246183 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.248239 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.248996 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.249114 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.249226 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.250007 (+ 781us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.250103 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3530,"thread_start_us":78,"threads_started":1}
I20260923 17:21:57.252990 5149 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7379eef1994eb5b96b7123ce212e41 (DEFAULT_TABLE table=test-table [id=fa9a5383d0014d6293c523ba6b5c5fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:57.254244 5149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7379eef1994eb5b96b7123ce212e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:57.256726 4867 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7379eef1994eb5b96b7123ce212e41 (DEFAULT_TABLE table=test-table [id=fa9a5383d0014d6293c523ba6b5c5fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:57.257778 4867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7379eef1994eb5b96b7123ce212e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:57.260533 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.241020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:45023 (local address 127.0.0.1:44464)
0923 17:21:57.249788 (+ 8768us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.249802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:57.249893 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:57.255233 (+ 5340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:57.255236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:57.255254 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:57.255335 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.255344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.257622 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.257625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.258847 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.258858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.260127 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.260131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.260222 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.260247 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:21:57.260322 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:21:57.260395 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8717,"thread_start_us":89,"threads_started":1}
I20260923 17:21:57.260998 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.241391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.23.214.130:45023)
0923 17:21:57.242363 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.242371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.254609 (+ 12238us) server_negotiation.cc:407] Connection header received
0923 17:21:57.254700 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.254705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.254908 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.255072 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:57.255470 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.257488 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.258992 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.259794 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.259916 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.260038 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.260762 (+ 724us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.260862 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":63,"threads_started":1}
I20260923 17:21:57.263895 5003 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7379eef1994eb5b96b7123ce212e41 (DEFAULT_TABLE table=test-table [id=fa9a5383d0014d6293c523ba6b5c5fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:21:57.264969 5003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7379eef1994eb5b96b7123ce212e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:57.270282 5242 tablet_bootstrap.cc:492] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: Bootstrap starting.
I20260923 17:21:57.272737 5242 tablet_bootstrap.cc:654] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:57.273672 5242 log.cc:824] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:57.274040 5245 tablet_bootstrap.cc:492] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Bootstrap starting.
I20260923 17:21:57.275894 5242 tablet_bootstrap.cc:492] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: No bootstrap required, opened a new log
I20260923 17:21:57.276089 5242 ts_tablet_manager.cc:1397] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:21:57.276598 5245 tablet_bootstrap.cc:654] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:57.277594 5245 log.cc:824] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:57.278304 5243 tablet_bootstrap.cc:492] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4: Bootstrap starting.
I20260923 17:21:57.280246 5242 raft_consensus.cc:348] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.280506 5242 raft_consensus.cc:374] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:57.280571 5242 raft_consensus.cc:729] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef499ca2d8904f99abe4aed83a6913de, State: Initialized, Role: FOLLOWER
I20260923 17:21:57.280970 5243 tablet_bootstrap.cc:654] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4: Neither blocks nor log segments found. Creating new log.
I20260923 17:21:57.282295 5243 log.cc:824] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:21:57.282269 5242 consensus_queue.cc:260] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.283365 5242 ts_tablet_manager.cc:1428] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:21:57.285298 5245 tablet_bootstrap.cc:492] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: No bootstrap required, opened a new log
I20260923 17:21:57.285528 5245 ts_tablet_manager.cc:1397] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260923 17:21:57.286055 5216 heartbeater.cc:503] Master 127.23.214.190:44811 was elected leader, sending a full tablet report...
I20260923 17:21:57.291263 5245 raft_consensus.cc:348] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.291647 5245 raft_consensus.cc:374] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:57.291723 5245 raft_consensus.cc:729] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc717344dd9a4d59a7d48c8a0a015e6d, State: Initialized, Role: FOLLOWER
I20260923 17:21:57.292255 5245 consensus_queue.cc:260] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.294016 5245 ts_tablet_manager.cc:1428] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260923 17:21:57.294425 5075 heartbeater.cc:503] Master 127.23.214.190:44811 was elected leader, sending a full tablet report...
I20260923 17:21:57.295015 5243 tablet_bootstrap.cc:492] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4: No bootstrap required, opened a new log
I20260923 17:21:57.295248 5243 ts_tablet_manager.cc:1397] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260923 17:21:57.299167 5243 raft_consensus.cc:348] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.299437 5243 raft_consensus.cc:374] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:21:57.299489 5243 raft_consensus.cc:729] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fc17e4ed184857b4f6bd5955d4dfd4, State: Initialized, Role: FOLLOWER
I20260923 17:21:57.300019 5243 consensus_queue.cc:260] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.300725 4939 heartbeater.cc:503] Master 127.23.214.190:44811 was elected leader, sending a full tablet report...
I20260923 17:21:57.301086 5243 ts_tablet_manager.cc:1428] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:21:57.345278 5250 raft_consensus.cc:482] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:21:57.345539 5250 raft_consensus.cc:504] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.347020 5250 leader_election.cc:288] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67fc17e4ed184857b4f6bd5955d4dfd4 (127.23.214.129:43749), ef499ca2d8904f99abe4aed83a6913de (127.23.214.131:40165)
I20260923 17:21:57.352487 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.347736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:55655 (local address 127.23.214.129:43749)
0923 17:21:57.347853 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.347857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.348061 (+ 204us) server_negotiation.cc:407] Connection header received
0923 17:21:57.348273 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.348276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.348321 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.348364 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:57.348826 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.349583 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.351272 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.351832 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.351921 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.351972 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.352287 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.352362 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:21:57.352684 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.347706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:49207 (local address 127.23.214.131:40165)
0923 17:21:57.347830 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.347835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.348148 (+ 313us) server_negotiation.cc:407] Connection header received
0923 17:21:57.348330 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.348333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.348380 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.348426 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:21:57.348888 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.349690 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.351352 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.351993 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.352102 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.352149 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.352482 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.352560 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:21:57.352850 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.347285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43749 (local address 127.23.214.130:55655)
0923 17:21:57.347976 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.347989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:57.348149 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:57.348491 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:57.348494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:57.348508 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:57.348705 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.348711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.349743 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.349746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.351135 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.351145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.352059 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.352062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.352162 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.352183 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:21:57.352622 (+ 439us) client_negotiation.cc:240] Negotiation successful
0923 17:21:57.352692 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":122,"spinlock_wait_cycles":36608,"thread_start_us":112,"threads_started":1}
I20260923 17:21:57.353200 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.347336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:40165 (local address 127.23.214.130:49207)
0923 17:21:57.348061 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.348075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:57.348217 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:57.348550 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:57.348553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:57.348583 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:21:57.348741 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.348751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.349837 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.349840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.351205 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:57.351217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.352243 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.352247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:57.352351 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.352372 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:21:57.352979 (+ 607us) client_negotiation.cc:240] Negotiation successful
0923 17:21:57.353061 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":12,"thread_start_us":108,"threads_started":1}
I20260923 17:21:57.353837 4892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7379eef1994eb5b96b7123ce212e41" candidate_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" is_pre_election: true
I20260923 17:21:57.353950 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7379eef1994eb5b96b7123ce212e41" candidate_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef499ca2d8904f99abe4aed83a6913de" is_pre_election: true
I20260923 17:21:57.354179 4892 raft_consensus.cc:2454] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc717344dd9a4d59a7d48c8a0a015e6d in term 0.
I20260923 17:21:57.354244 5170 raft_consensus.cc:2454] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc717344dd9a4d59a7d48c8a0a015e6d in term 0.
I20260923 17:21:57.354688 4964 leader_election.cc:302] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67fc17e4ed184857b4f6bd5955d4dfd4, ef499ca2d8904f99abe4aed83a6913de, fc717344dd9a4d59a7d48c8a0a015e6d; no voters:
I20260923 17:21:57.354988 5250 raft_consensus.cc:2781] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:21:57.355086 5250 raft_consensus.cc:482] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:21:57.355131 5250 raft_consensus.cc:3037] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:57.356442 5250 raft_consensus.cc:504] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
I20260923 17:21:57.356827 5250 leader_election.cc:288] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [CANDIDATE]: Term 1 election: Requested vote from peers 67fc17e4ed184857b4f6bd5955d4dfd4 (127.23.214.129:43749), ef499ca2d8904f99abe4aed83a6913de (127.23.214.131:40165)
I20260923 17:21:57.357197 4892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7379eef1994eb5b96b7123ce212e41" candidate_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4"
I20260923 17:21:57.357219 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7379eef1994eb5b96b7123ce212e41" candidate_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef499ca2d8904f99abe4aed83a6913de"
I20260923 17:21:57.357311 4892 raft_consensus.cc:3037] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:57.357328 5170 raft_consensus.cc:3037] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:21:57.358530 4892 raft_consensus.cc:2454] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc717344dd9a4d59a7d48c8a0a015e6d in term 1.
I20260923 17:21:57.358644 5170 raft_consensus.cc:2454] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc717344dd9a4d59a7d48c8a0a015e6d in term 1.
I20260923 17:21:57.358876 4964 leader_election.cc:302] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67fc17e4ed184857b4f6bd5955d4dfd4, fc717344dd9a4d59a7d48c8a0a015e6d; no voters:
I20260923 17:21:57.359112 5250 raft_consensus.cc:2781] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:21:57.359316 5250 raft_consensus.cc:686] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 1 LEADER]: Becoming Leader. State: Replica: fc717344dd9a4d59a7d48c8a0a015e6d, State: Running, Role: LEADER
I20260923 17:21:57.359653 5250 consensus_queue.cc:237] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } }
W20260923 17:21:57.365321 4940 tablet.cc:2369] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:21:57.365334 4739 catalog_manager.cc:5495] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d reported cstate change: term changed from 0 to 1, leader changed from <none> to fc717344dd9a4d59a7d48c8a0a015e6d (127.23.214.130). New cstate: current_term: 1 leader_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc717344dd9a4d59a7d48c8a0a015e6d" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45023 } health_report { overall_health: HEALTHY } } }
I20260923 17:21:57.406502 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.401315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.23.214.130:45023)
0923 17:21:57.401457 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:57.401463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:57.401479 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:21:57.401566 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:57.401569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:57.401648 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:57.401764 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:21:57.402190 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.402973 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.404258 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:57.404565 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:57.404688 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:57.405108 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:21:57.405679 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:21:57.405798 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:57.406276 (+ 478us) server_negotiation.cc:299] Negotiation successful
0923 17:21:57.406339 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:21:57.413393 5170 raft_consensus.cc:1261] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 1 FOLLOWER]: Refusing update from remote peer fc717344dd9a4d59a7d48c8a0a015e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:57.413393 4892 raft_consensus.cc:1261] T 7d7379eef1994eb5b96b7123ce212e41 P 67fc17e4ed184857b4f6bd5955d4dfd4 [term 1 FOLLOWER]: Refusing update from remote peer fc717344dd9a4d59a7d48c8a0a015e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:21:57.414127 5250 consensus_queue.cc:1035] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67fc17e4ed184857b4f6bd5955d4dfd4" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:21:57.414290 5254 consensus_queue.cc:1035] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef499ca2d8904f99abe4aed83a6913de" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:21:57.433411 5217 tablet.cc:2369] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 17:21:57 dist-test-slave-9q7f krb5kdc[3738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for HTTP/127.23.214.148@KRBTEST.COM
I20260923 17:21:57.436533 5257 mvcc.cc:205] Tried to move back new op lower bound from 7332594144917327872 to 7332594144708067328. Current Snapshot: MvccSnapshot[applied={T|T < 7332594144917327872}]
I20260923 17:21:57.436749 5260 mvcc.cc:205] Tried to move back new op lower bound from 7332594144917327872 to 7332594144708067328. Current Snapshot: MvccSnapshot[applied={T|T < 7332594144917327872}]
I20260923 17:21:57.445972 5259 mvcc.cc:205] Tried to move back new op lower bound from 7332594144917327872 to 7332594144708067328. Current Snapshot: MvccSnapshot[applied={T|T < 7332594144917327872}]
I20260923 17:21:57.448757 4739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:57.448935 4739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:21:57.451834 4739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d6db39000464958a784845f93553523: Sending DeleteTablet for 3 replicas of tablet 7d7379eef1994eb5b96b7123ce212e41
I20260923 17:21:57.452921 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7379eef1994eb5b96b7123ce212e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:57 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:55330
I20260923 17:21:57.453056 4867 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7379eef1994eb5b96b7123ce212e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:57 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:59776
I20260923 17:21:57.453519 5003 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7379eef1994eb5b96b7123ce212e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:21:57 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:44464
I20260923 17:21:57.454874 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 4777
I20260923 17:21:57.459789 5270 tablet_replica.cc:330] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: stopping tablet replica
I20260923 17:21:57.460196 5270 raft_consensus.cc:2229] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:21:57.466212 5270 raft_consensus.cc:2258] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:57.468255 5270 ts_tablet_manager.cc:1905] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:21:57.471228 5270 ts_tablet_manager.cc:1918] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:21:57.471405 5270 log.cc:1196] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/7d7379eef1994eb5b96b7123ce212e41
I20260923 17:21:57.471844 5270 ts_tablet_manager.cc:1939] T 7d7379eef1994eb5b96b7123ce212e41 P fc717344dd9a4d59a7d48c8a0a015e6d: Deleting consensus metadata
W20260923 17:21:57.472175 4726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:43749 (error 108)
I20260923 17:21:57.473172 5268 tablet_replica.cc:330] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de: stopping tablet replica
I20260923 17:21:57.473759 5268 raft_consensus.cc:2229] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:21:57.474287 4725 catalog_manager.cc:4841] TS fc717344dd9a4d59a7d48c8a0a015e6d (127.23.214.130:45023): tablet 7d7379eef1994eb5b96b7123ce212e41 (table test-table [id=fa9a5383d0014d6293c523ba6b5c5fda]) successfully deleted
I20260923 17:21:57.474671 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 4943
I20260923 17:21:57.477416 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:57.476996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:43749 (local address 127.0.0.1:59778)
0923 17:21:57.477212 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:57.477308 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:43749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260923 17:21:57.477638 4726 catalog_manager.cc:4584] TS 67fc17e4ed184857b4f6bd5955d4dfd4 (127.23.214.129:43749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d7379eef1994eb5b96b7123ce212e41: Network error: Client connection negotiation failed: client connection to 127.23.214.129:43749: connect: Connection refused (error 111)
I20260923 17:21:57.481554 5268 raft_consensus.cc:2258] T 7d7379eef1994eb5b96b7123ce212e41 P ef499ca2d8904f99abe4aed83a6913de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:21:57.483342 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5078
I20260923 17:21:57.490837 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 4706
I20260923 17:21:57.499739 24410 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:21:57.845286 24410 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:21:57.845431 24410 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:21:58.181002 24410 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:21:58.181 UTC [3768] LOG: received smart shutdown request
2026-09-23 17:21:58.185 UTC [3768] LOG: background worker "logical replication launcher" (PID 3777) exited with exit code 1
2026-09-23 17:21:58.185 UTC [3772] LOG: shutting down
2026-09-23 17:21:58.360 UTC [3768] LOG: database system is shut down
2026-09-23T17:21:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49440 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:21:58 dist-test-slave-9q7f krb5kdc[5281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:21:58 dist-test-slave-9q7f krb5kdc[5281](info): set up 2 sockets
Sep 23 17:21:58 dist-test-slave-9q7f krb5kdc[5281](info): commencing operation
krb5kdc: starting...
W20260923 17:21:59.528210 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:21:59 dist-test-slave-9q7f krb5kdc[5281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:21:59.665030 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:46189
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:46189,127.23.214.189:34707,127.23.214.188:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:21:59.777045 5294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:21:59.777336 5294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:21:59.777379 5294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:21:59.781613 5294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:21:59.781679 5294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:21:59.781698 5294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:21:59.781715 5294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:21:59.781731 5294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:21:59.787177 5294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:46189,127.23.214.189:34707,127.23.214.188:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:46189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:21:59.788298 5294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:21:59.789345 5294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:21:59.798441 5299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:59.798441 5300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:21:59.798444 5302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:21:59.798635 5294 server_base.cc:1034] running on GCE node
I20260923 17:21:59.799366 5294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:21:59.800519 5294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:21:59.801687 5294 hybrid_clock.cc:648] HybridClock initialized: now 1790184119801657 us; error 40 us; skew 500 ppm
Sep 23 17:21:59 dist-test-slave-9q7f krb5kdc[5281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:21:59.804492 5294 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:21:59.805439 5294 webserver.cc:460] Webserver started at http://127.23.214.190:39571/ using document root <none> and password file <none>
I20260923 17:21:59.806093 5294 fs_manager.cc:359] Metadata directory not provided
I20260923 17:21:59.806190 5294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:21:59.806430 5294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:21:59.808547 5294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "bda6f2e84a1d418bb06d503182a741ff"
format_stamp: "Formatted at 2026-09-23 17:21:59 on dist-test-slave-9q7f"
server_key: "272f4b43095c66bf5f71dbfb05676579"
server_key_iv: "55a3fef3b5e87efc61dc5b90d0f5d27f"
server_key_version: "encryptionkey@0"
I20260923 17:21:59.809155 5294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "bda6f2e84a1d418bb06d503182a741ff"
format_stamp: "Formatted at 2026-09-23 17:21:59 on dist-test-slave-9q7f"
server_key: "272f4b43095c66bf5f71dbfb05676579"
server_key_iv: "55a3fef3b5e87efc61dc5b90d0f5d27f"
server_key_version: "encryptionkey@0"
I20260923 17:21:59.813174 5294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 17:21:59.815796 5309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:21:59.817054 5294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:21:59.817196 5294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "bda6f2e84a1d418bb06d503182a741ff"
format_stamp: "Formatted at 2026-09-23 17:21:59 on dist-test-slave-9q7f"
server_key: "272f4b43095c66bf5f71dbfb05676579"
server_key_iv: "55a3fef3b5e87efc61dc5b90d0f5d27f"
server_key_version: "encryptionkey@0"
I20260923 17:21:59.817330 5294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:21:59.844693 5294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:21:59.848516 5294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:21:59.848763 5294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:21:59.856877 5294 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:46189
I20260923 17:21:59.857234 5361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:46189 every 8 connection(s)
I20260923 17:21:59.857949 5294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:21:59.860809 5362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:21:59.862082 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5294
I20260923 17:21:59.862231 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:21:59.862507 24410 external_mini_cluster.cc:1428] Setting key 0d05616923764c95755bf1d12f4d4f53
I20260923 17:21:59.864807 5362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.214.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.189@KRBTEST.COM" created.
Sep 23 17:21:59 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:21:59.881070 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.867264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.23.214.190:46189)
0923 17:21:59.867731 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 17:21:59.867750 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:21:59.868028 (+ 278us) server_negotiation.cc:407] Connection header received
0923 17:21:59.869041 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:21:59.869045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:21:59.869217 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:21:59.869406 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:21:59.870320 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.871231 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.872330 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.872737 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.873021 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.875840 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:21:59.875847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:21:59.875858 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:21:59.875882 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:21:59.877809 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:59.878480 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:59.878485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:59.878487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:59.878594 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:21:59.878970 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:21:59.878972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:21:59.878973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:21:59.879371 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:21:59.879476 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:21:59.879977 (+ 501us) server_negotiation.cc:299] Negotiation successful
0923 17:21:59.880225 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":311,"thread_start_us":154,"threads_started":1}
I20260923 17:21:59.881074 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.867264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:46189 (local address 127.0.0.1:38460)
0923 17:21:59.867871 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:59.867908 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 17:21:59.868736 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:21:59.869399 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:21:59.869404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:21:59.869814 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:21:59.870163 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:59.870201 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.871361 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.871364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:59.872212 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:21:59.872221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:21:59.872891 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:21:59.872894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:21:59.873114 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:21:59.873760 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:21:59.873777 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:21:59.875645 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:21:59.877986 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:59.877999 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:59.878012 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:59.878360 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:59.878697 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:21:59.878703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:21:59.878706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:21:59.878874 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:21:59.879508 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:21:59.879515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:21:59.879698 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 17:21:59.879912 (+ 214us) client_negotiation.cc:240] Negotiation successful
0923 17:21:59.880338 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":162,"thread_start_us":163,"threads_started":1}
I20260923 17:21:59.883612 5362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:21:59.884349 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.884104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57642)
0923 17:21:59.884219 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:59.884289 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:21:59.884546 5311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
I20260923 17:21:59.885663 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.885283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57644)
0923 17:21:59.885407 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:59.885549 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:21:59.885836 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:34707: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab.
Entry for principal kudu/127.23.214.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab.
I20260923 17:21:59.889067 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:34707
--webserver_interface=127.23.214.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab
--principal=kudu/127.23.214.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:46189,127.23.214.189:34707,127.23.214.188:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:21:59.935122 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } attempt: 1
I20260923 17:21:59.936064 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.935743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57658)
0923 17:21:59.935904 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:59.935973 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:21:59.936813 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:21:59.936601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57666)
0923 17:21:59.936709 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 17:21:59.936760 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:21:59.936942 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:34707: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
W20260923 17:22:00.003057 5369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:00.003374 5369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:00.003458 5369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:22:00.005105 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } attempt: 2
I20260923 17:22:00.006006 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.005683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57674)
0923 17:22:00.005825 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.005917 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 17:22:00.006711 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.006496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57680)
0923 17:22:00.006605 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.006652 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:22:00.006839 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:34707: Network error: Client connection negotiation failed: client connection to 127.23.214.189:34707: connect: Connection refused (error 111)
W20260923 17:22:00.007961 5369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:00.008031 5369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:00.008076 5369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:00.008117 5369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:00.008152 5369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:00.013530 5369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:46189,127.23.214.189:34707,127.23.214.188:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab
--principal=kudu/127.23.214.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.214.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:00.014782 5369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:00.015820 5369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:00.024434 5374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:00.024426 5376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:00.024425 5378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:00.024848 5369 server_base.cc:1034] running on GCE node
I20260923 17:22:00.025431 5369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:00.026767 5369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:00.027951 5369 hybrid_clock.cc:648] HybridClock initialized: now 1790184120027933 us; error 29 us; skew 500 ppm
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:00.030926 5369 init.cc:377] Logged in from keytab as kudu/127.23.214.189@KRBTEST.COM (short username kudu)
I20260923 17:22:00.031850 5369 webserver.cc:460] Webserver started at http://127.23.214.189:40057/ using document root <none> and password file <none>
I20260923 17:22:00.032425 5369 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:00.032519 5369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:00.032763 5369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:00.034996 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data/instance:
uuid: "5ec440819ceb4908bfba3064433ef9b3"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-9q7f"
server_key: "45d16233f6b4ca3cbf0004ffa2fc3378"
server_key_iv: "90cb76af05949c4b28001655b8b1fde4"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.035604 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/wal/instance:
uuid: "5ec440819ceb4908bfba3064433ef9b3"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-9q7f"
server_key: "45d16233f6b4ca3cbf0004ffa2fc3378"
server_key_iv: "90cb76af05949c4b28001655b8b1fde4"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.039430 5369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:22:00.042038 5385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:00.043383 5369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:00.043578 5369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/wal
uuid: "5ec440819ceb4908bfba3064433ef9b3"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-9q7f"
server_key: "45d16233f6b4ca3cbf0004ffa2fc3378"
server_key_iv: "90cb76af05949c4b28001655b8b1fde4"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.043704 5369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:00.063179 5369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:00.066435 5369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:00.066650 5369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:00.073606 5369 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.189:34707
I20260923 17:22:00.073916 5437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.189:34707 every 8 connection(s)
I20260923 17:22:00.074666 5369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
I20260923 17:22:00.075474 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5369
I20260923 17:22:00.075611 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-1/wal/instance
I20260923 17:22:00.075876 24410 external_mini_cluster.cc:1428] Setting key 6ffb4819dc9ee016952a2ed588d61952
I20260923 17:22:00.077739 5438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:00.083325 5438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.214.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.188@KRBTEST.COM" created.
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:00.097958 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } attempt: 3
I20260923 17:22:00.099015 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.085936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.23.214.190:46189)
0923 17:22:00.086070 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.086075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.086963 (+ 888us) server_negotiation.cc:407] Connection header received
0923 17:22:00.088110 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.088116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.088187 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.088289 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.089442 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.090021 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.091225 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.091475 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.091591 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.094318 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.094323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.094326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.094359 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.096465 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.097070 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.097073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.097074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.097218 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.097634 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.097637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.097639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.097853 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.097992 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.098694 (+ 702us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.098855 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:22:00.100097 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.086271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:46189 (local address 127.0.0.1:38462)
0923 17:22:00.086812 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.086855 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.087882 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.088435 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.088441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.088794 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.089271 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.089285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.090200 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.090202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.091092 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.091102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.091653 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.091657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.091894 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.092571 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.092595 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.094112 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.096661 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.096666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.096681 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.096972 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.097331 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.097336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.097339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.097499 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.098012 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.098020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.098389 (+ 369us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.098644 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.099049 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":114,"threads_started":1}
I20260923 17:22:00.101785 5438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab.
Entry for principal kudu/127.23.214.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab.
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:22:00.105281 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:35723
--webserver_interface=127.23.214.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab
--principal=kudu/127.23.214.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:46189,127.23.214.189:34707,127.23.214.188:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:22:00.109383 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.098453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57690)
0923 17:22:00.099045 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.099063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.099152 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.099717 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.099720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.099889 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.100001 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.100007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.101489 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.101492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.102325 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.102337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.103107 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.103110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.103193 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.103713 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.103727 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.105195 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.107473 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.107479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.107484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.107771 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.108115 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.108119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.108121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.108206 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.108703 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.108709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.108830 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.108940 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.109111 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507}
I20260923 17:22:00.110324 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.098591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.23.214.189:34707)
0923 17:22:00.099038 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.099053 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.099318 (+ 265us) server_negotiation.cc:407] Connection header received
0923 17:22:00.099372 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.099381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.099589 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.099698 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.100119 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.101330 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.102516 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.103004 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.103124 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.105344 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.105353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.105386 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.105419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.107359 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.107909 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.107917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.107919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.108007 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.108299 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.108304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.108306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.108587 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.108733 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.109820 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.110025 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260923 17:22:00.112345 5362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:22:00.113297 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.102349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57694)
0923 17:22:00.102540 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.102552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.102607 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.102992 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.102994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.103172 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.103281 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.103286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.104377 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.104380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.105519 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.105542 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.106095 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.106102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.106310 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.107478 (+ 1168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.107488 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.108842 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.111673 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.111676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.111679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.111911 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.112245 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.112250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.112254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.112344 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.112793 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.112796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.112881 (+ 85us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.112985 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.113126 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":36}
I20260923 17:22:00.113307 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.112939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34560)
0923 17:22:00.113142 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.113197 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260923 17:22:00.113348 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.102468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.23.214.189:34707)
0923 17:22:00.102798 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.102805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.102814 (+ 9us) server_negotiation.cc:407] Connection header received
0923 17:22:00.102839 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.102841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.102870 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.102973 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.103399 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.104262 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.105651 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.105987 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.106101 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.108957 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.108961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.108963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.108982 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.111533 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.112017 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.112024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.112029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.112128 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.112445 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.112450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.112455 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.112658 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.112793 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.113037 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.113202 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260923 17:22:00.114094 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.113808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34566)
0923 17:22:00.113984 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.114033 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260923 17:22:00.114255 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:35723: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
I20260923 17:22:00.114547 5438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:22:00.115321 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.114999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34582)
0923 17:22:00.115142 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.115234 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:22:00.115577 5387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
I20260923 17:22:00.116505 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.116232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34596)
0923 17:22:00.116369 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.116430 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:22:00.116681 5438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:35723: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
I20260923 17:22:00.145444 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } attempt: 1
I20260923 17:22:00.146446 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.146102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34602)
0923 17:22:00.146276 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.146351 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:22:00.147301 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.147057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34606)
0923 17:22:00.147176 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.147231 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:22:00.147475 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:35723: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
I20260923 17:22:00.165990 5438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } attempt: 1
I20260923 17:22:00.166955 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.166644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34614)
0923 17:22:00.166810 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.166876 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:22:00.167819 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.167627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34624)
0923 17:22:00.167719 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.167765 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 17:22:00.168011 5438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:35723: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
I20260923 17:22:00.222640 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } attempt: 2
W20260923 17:22:00.222640 5446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:00.222982 5446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:00.223063 5446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:22:00.223453 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.223148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34628)
0923 17:22:00.223305 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.223371 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:22:00.224205 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.223993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34638)
0923 17:22:00.224102 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.224149 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:22:00.224356 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:35723: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
W20260923 17:22:00.227591 5446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:00.227640 5446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:00.227660 5446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:00.227699 5446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:00.227738 5446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:00.233189 5446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:46189,127.23.214.189:34707,127.23.214.188:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab
--principal=kudu/127.23.214.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.214.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:00.234300 5446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:00.235313 5446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:22:00.236162 5438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } attempt: 2
I20260923 17:22:00.236897 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.236642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34642)
0923 17:22:00.236777 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.236831 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 17:22:00.237607 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.237389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34658)
0923 17:22:00.237495 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.237551 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:22:00.237721 5438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:35723: Network error: Client connection negotiation failed: client connection to 127.23.214.188:35723: connect: Connection refused (error 111)
W20260923 17:22:00.244150 5452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:00.244174 5455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:00.244103 5453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:00.244287 5446 server_base.cc:1034] running on GCE node
I20260923 17:22:00.244979 5446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:00.246178 5446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:00.247364 5446 hybrid_clock.cc:648] HybridClock initialized: now 1790184120247351 us; error 30 us; skew 500 ppm
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:00.250208 5446 init.cc:377] Logged in from keytab as kudu/127.23.214.188@KRBTEST.COM (short username kudu)
I20260923 17:22:00.251058 5446 webserver.cc:460] Webserver started at http://127.23.214.188:41915/ using document root <none> and password file <none>
I20260923 17:22:00.251626 5446 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:00.251731 5446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:00.251911 5446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:00.253859 5446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data/instance:
uuid: "e4525c5c20ad44d7b67c66fc3b740c99"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-9q7f"
server_key: "e16edef8900ffee43acb7efa03686b24"
server_key_iv: "fc7e3d108526c09713095fc8772a1404"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.254436 5446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/wal/instance:
uuid: "e4525c5c20ad44d7b67c66fc3b740c99"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-9q7f"
server_key: "e16edef8900ffee43acb7efa03686b24"
server_key_iv: "fc7e3d108526c09713095fc8772a1404"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.258201 5446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:22:00.260536 5462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:00.261772 5446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:00.261971 5446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/wal
uuid: "e4525c5c20ad44d7b67c66fc3b740c99"
format_stamp: "Formatted at 2026-09-23 17:22:00 on dist-test-slave-9q7f"
server_key: "e16edef8900ffee43acb7efa03686b24"
server_key_iv: "fc7e3d108526c09713095fc8772a1404"
server_key_version: "encryptionkey@0"
I20260923 17:22:00.262090 5446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:00.289361 5446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:00.292443 5446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:00.292644 5446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:00.299502 5446 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.188:35723
I20260923 17:22:00.299806 5515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.188:35723 every 8 connection(s)
I20260923 17:22:00.300544 5446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
I20260923 17:22:00.302650 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5446
I20260923 17:22:00.302785 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790183952943234-24410-0/minicluster-data/master-2/wal/instance
I20260923 17:22:00.303083 24410 external_mini_cluster.cc:1428] Setting key cb44f4d2ba25d4ce10e154d02942410e
I20260923 17:22:00.303488 5516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:00.308178 5516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:00.318085 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.304459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.23.214.188:35723)
0923 17:22:00.304863 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.304879 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.304913 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:22:00.305560 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.305579 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.305970 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.306253 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.307189 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.308173 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.309295 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.309820 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.310032 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.312713 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.312719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.312728 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.312755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.314687 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.315168 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.315172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.315176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.315310 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.315585 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.315588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.315591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.316104 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.316240 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.316624 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.316927 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260923 17:22:00.319829 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.310349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.23.214.190:46189)
0923 17:22:00.310519 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.310525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.310898 (+ 373us) server_negotiation.cc:407] Connection header received
0923 17:22:00.311067 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.311070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.311121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.311208 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.311998 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.312557 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.313471 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.313806 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.314026 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.316496 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.316500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.316503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.316528 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.318118 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.318623 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.318627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.318629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.318715 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.318963 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.318966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.318967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.319195 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.319290 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.319627 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.319720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:22:00.319895 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.310098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:46189 (local address 127.0.0.1:38464)
0923 17:22:00.310781 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.310819 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.310958 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.311258 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.311263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.311687 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.311855 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.311865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.312678 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.312681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.313322 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.313348 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.314043 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.314047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.314137 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.314614 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.314626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.316344 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.318235 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.318246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.318255 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.318525 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.318806 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.318808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.318809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.318881 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.319310 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.319316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.319427 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.319571 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.319736 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":132,"threads_started":1}
I20260923 17:22:00.321532 5516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:22:00.323549 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } attempt: 3
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
I20260923 17:22:00.328897 5438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } attempt: 3
I20260923 17:22:00.332000 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.322065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:34707 (local address 127.0.0.1:57704)
0923 17:22:00.322170 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.322185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.322270 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.322562 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.322565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.322741 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.322826 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.322831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.323558 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.323563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.324497 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.324503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.325665 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.325669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.325767 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.326226 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.326239 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.327631 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.330327 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.330330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.330332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.330521 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.330852 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.330858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.330861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.330950 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.331446 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.331451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.331546 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.331653 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.331792 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 17:22:00.332175 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.321094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.23.214.190:46189)
0923 17:22:00.321199 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.321204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.321324 (+ 120us) server_negotiation.cc:407] Connection header received
0923 17:22:00.321474 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.321477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.321515 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.321600 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.322393 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.322896 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.323748 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.324070 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.324169 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.327484 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.327488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.327491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.327517 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.329659 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.330164 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.330183 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.330187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.330284 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.330625 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.330631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.330636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.330860 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.331101 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.331835 (+ 734us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.331985 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 17:22:00.332580 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.322210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.23.214.189:34707)
0923 17:22:00.322337 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.322343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.322358 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:22:00.322402 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.322405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.322456 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.322546 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.322950 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.323452 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.324963 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.325382 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.325505 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.328148 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.328154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.328157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.328191 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.330243 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.330634 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.330638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.330640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.330736 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.331070 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.331075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.331077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.331315 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.331403 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.332304 (+ 901us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.332436 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:22:00.332932 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.323884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34682)
0923 17:22:00.324022 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.324035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.324103 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.324318 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.324320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.324492 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.324608 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.324615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.325390 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.325394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.325986 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.325997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.326597 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.326600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.326714 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.327195 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.327210 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.328484 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.331402 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.331405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.331408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.331660 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.331944 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.331947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.331949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.332038 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.332462 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.332466 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.332553 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.332633 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.332778 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:22:00.333372 5516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:22:00.333477 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.323969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.23.214.188:35723)
0923 17:22:00.324079 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.324083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.324096 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:22:00.324176 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.324178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.324213 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.324287 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.324718 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.325283 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.326091 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.326463 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.326568 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.328593 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.328597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.328601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.328629 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.331279 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.331772 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.331776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.331777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.331846 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.332157 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.332161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.332163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.332351 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.333122 (+ 771us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.333194 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.333295 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:22:00.336983 5362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff: Bootstrap starting.
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184120, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
I20260923 17:22:00.340190 5362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff: Neither blocks nor log segments found. Creating new log.
Sep 23 17:22:00 dist-test-slave-9q7f krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:22:00.341202 5362 log.cc:824] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:00.342934 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.333846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34706)
0923 17:22:00.334089 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.334103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.334175 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.334420 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.334425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.334590 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.334697 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.334705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.335661 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.335664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.336484 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.336494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.336989 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.336992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.337064 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.337497 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.337517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.339049 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.341272 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.341275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.341277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.341515 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.341809 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.341812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.341814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.341924 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.342387 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.342390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.342476 (+ 86us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.342574 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.342723 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":4,"spinlock_wait_cycles":1920}
I20260923 17:22:00.343312 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.329343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:35723 (local address 127.0.0.1:34692)
0923 17:22:00.329445 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:00.329461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:00.329538 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:00.330670 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:00.330676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:00.330849 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:00.330965 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.330972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.334479 (+ 3507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.334482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.335240 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:00.335251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.335948 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.335952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:00.336030 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.336521 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:00.336551 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:00.338132 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:00.341161 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.341165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.341167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.341970 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.342288 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:00.342291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:00.342293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:00.342380 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:00.342814 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:00.342817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:00.342921 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 17:22:00.343020 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:22:00.343164 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 17:22:00.343554 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.333908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.23.214.188:35723)
0923 17:22:00.333996 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.334000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.334245 (+ 245us) server_negotiation.cc:407] Connection header received
0923 17:22:00.334272 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.334274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.334310 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.334376 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.334821 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.335561 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.336602 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.336891 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.337333 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.339213 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.339217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.339220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.339249 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.341162 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.341633 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.341636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.341638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.341715 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.342031 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.342037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.342039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.342283 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.343226 (+ 943us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.343307 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.343415 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:22:00.343647 5362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff: No bootstrap required, opened a new log
I20260923 17:22:00.343879 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.329400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.23.214.188:35723)
0923 17:22:00.330412 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.330421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.330434 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:22:00.330468 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.330470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.330513 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.331080 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.331173 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.333601 (+ 2428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.335386 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.335812 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.337162 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.338283 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.338288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.338294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.338323 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.341029 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.342063 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.342068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.342072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.342169 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.342477 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.342480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.342482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.342685 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.342770 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.343618 (+ 848us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.343730 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":920,"spinlock_wait_cycles":896,"thread_start_us":117,"threads_started":1}
I20260923 17:22:00.345338 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.333516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.23.214.189:34707)
0923 17:22:00.333608 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.333612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.333860 (+ 248us) server_negotiation.cc:407] Connection header received
0923 17:22:00.334090 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.334095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.334141 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.334215 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.335041 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.335734 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.336771 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.337086 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.337923 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.340971 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.340977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.340982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.341013 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.342738 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.343897 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.343901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.343903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.344003 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.344344 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.344350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.344352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.344553 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.344704 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.345090 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.345211 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:22:00.346810 5438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: Bootstrap starting.
I20260923 17:22:00.346843 5516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: Bootstrap starting.
I20260923 17:22:00.348043 5362 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.348275 5362 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:00.348330 5362 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda6f2e84a1d418bb06d503182a741ff, State: Initialized, Role: FOLLOWER
I20260923 17:22:00.348800 5362 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.349282 5516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:00.349279 5438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:00.349319 5523 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.349442 5523 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:00.350082 5438 log.cc:824] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:00.350114 5516 log.cc:824] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:00.350232 5362 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:22:00.350625 5315 tablet.cc:2369] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:00.352762 5438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: No bootstrap required, opened a new log
I20260923 17:22:00.354112 5516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: No bootstrap required, opened a new log
I20260923 17:22:00.356791 5438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.357009 5438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:00.357072 5438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec440819ceb4908bfba3064433ef9b3, State: Initialized, Role: FOLLOWER
I20260923 17:22:00.357265 5516 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.357466 5516 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:00.357542 5516 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4525c5c20ad44d7b67c66fc3b740c99, State: Initialized, Role: FOLLOWER
I20260923 17:22:00.358086 5516 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.358947 5438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.359864 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:22:00.360260 5438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:22:00.355363 5536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:22:00.362022 5536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:22:00.360502 5294 master_runner.cc:186] Error getting master registration for 127.23.214.189:34707: OK, instance_id { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" instance_seqno: 1790184120071756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:22:00.364179 5294 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:22:00.365454 5516 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:22:00.372385 5554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:22:00.372480 5554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:22:00.372851 5369 master_runner.cc:263] No leader master found from master 5ec440819ceb4908bfba3064433ef9b3
I20260923 17:22:00.372925 5369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:22:00.373900 5446 master_runner.cc:263] No leader master found from master e4525c5c20ad44d7b67c66fc3b740c99
I20260923 17:22:00.373970 5446 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:22:00.374140 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.374264 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:00.378369 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.362532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.23.214.190:46189)
0923 17:22:00.362641 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.362645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.364616 (+ 1971us) server_negotiation.cc:407] Connection header received
0923 17:22:00.364799 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.364803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.364932 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.364987 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.366178 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.366972 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.368888 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.369220 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.369294 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.374278 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.374283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.374286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.374321 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.376272 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.376829 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.376833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.376835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.376930 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.377276 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.377280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.377282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.377514 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.377621 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.378025 (+ 404us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.378175 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260923 17:22:00.380896 5570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:22:00.380993 5570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:22:00.382263 5538 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.382411 5538 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:00.388118 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.363144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.23.214.189:34707)
0923 17:22:00.363249 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.363253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.370737 (+ 7484us) server_negotiation.cc:407] Connection header received
0923 17:22:00.370917 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.370920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.370970 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.371025 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.378305 (+ 7280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.379086 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.381147 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.381474 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.381592 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.383594 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.383599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.383602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.383639 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.385362 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.386692 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.386697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.386699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.386787 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.387101 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.387105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.387107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.387331 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.387448 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.387801 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.387930 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:22:00.392815 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:00.363800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.23.214.188:35723)
0923 17:22:00.363911 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:00.363915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:00.378512 (+ 14597us) server_negotiation.cc:407] Connection header received
0923 17:22:00.378562 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:00.378565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:00.378626 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:00.378700 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:00.381351 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.382392 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.383657 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:00.384340 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:00.384681 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:00.388939 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:00.388943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:00.388946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:00.388975 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:00.390898 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.391409 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.391413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.391415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.391502 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:00.391811 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:00.391815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:00.391817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:00.392026 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:00.392141 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:00.392524 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 17:22:00.392650 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 17:22:00.443842 5523 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:00.444052 5523 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.445112 5523 leader_election.cc:288] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec440819ceb4908bfba3064433ef9b3 (127.23.214.189:34707), e4525c5c20ad44d7b67c66fc3b740c99 (127.23.214.188:35723)
I20260923 17:22:00.446048 5489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bda6f2e84a1d418bb06d503182a741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" is_pre_election: true
I20260923 17:22:00.446388 5489 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bda6f2e84a1d418bb06d503182a741ff in term 0.
I20260923 17:22:00.446944 5311 leader_election.cc:302] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bda6f2e84a1d418bb06d503182a741ff, e4525c5c20ad44d7b67c66fc3b740c99; no voters:
I20260923 17:22:00.447654 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bda6f2e84a1d418bb06d503182a741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec440819ceb4908bfba3064433ef9b3" is_pre_election: true
I20260923 17:22:00.447924 5412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bda6f2e84a1d418bb06d503182a741ff in term 0.
I20260923 17:22:00.448598 5523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:00.448684 5523 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:00.448726 5523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:00.450491 5523 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.451531 5489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bda6f2e84a1d418bb06d503182a741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4525c5c20ad44d7b67c66fc3b740c99"
I20260923 17:22:00.451663 5489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:00.453049 5489 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bda6f2e84a1d418bb06d503182a741ff in term 1.
I20260923 17:22:00.453531 5311 leader_election.cc:302] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bda6f2e84a1d418bb06d503182a741ff, e4525c5c20ad44d7b67c66fc3b740c99; no voters:
I20260923 17:22:00.454123 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bda6f2e84a1d418bb06d503182a741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec440819ceb4908bfba3064433ef9b3"
I20260923 17:22:00.454258 5412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:00.455559 5412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bda6f2e84a1d418bb06d503182a741ff in term 1.
I20260923 17:22:00.451023 5523 leader_election.cc:288] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [CANDIDATE]: Term 1 election: Requested vote from peers 5ec440819ceb4908bfba3064433ef9b3 (127.23.214.189:34707), e4525c5c20ad44d7b67c66fc3b740c99 (127.23.214.188:35723)
I20260923 17:22:00.456198 5523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:00.456327 5523 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [term 1 LEADER]: Becoming Leader. State: Replica: bda6f2e84a1d418bb06d503182a741ff, State: Running, Role: LEADER
I20260923 17:22:00.456722 5523 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } }
I20260923 17:22:00.462400 5523 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda6f2e84a1d418bb06d503182a741ff. Latest consensus state: current_term: 1 leader_uuid: "bda6f2e84a1d418bb06d503182a741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.462612 5523 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:00.463106 5574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:00.464102 5574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:00.474768 5489 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [term 1 FOLLOWER]: Refusing update from remote peer bda6f2e84a1d418bb06d503182a741ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:00.475663 5572 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:00.480433 5538 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda6f2e84a1d418bb06d503182a741ff. Latest consensus state: current_term: 1 leader_uuid: "bda6f2e84a1d418bb06d503182a741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.480587 5538 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:00.481845 5412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [term 1 FOLLOWER]: Refusing update from remote peer bda6f2e84a1d418bb06d503182a741ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:00.482627 5572 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:00.487103 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda6f2e84a1d418bb06d503182a741ff. Latest consensus state: current_term: 1 leader_uuid: "bda6f2e84a1d418bb06d503182a741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.487237 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:00.494527 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bda6f2e84a1d418bb06d503182a741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.494674 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:00.496150 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bda6f2e84a1d418bb06d503182a741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.496279 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:00.496774 5575 mvcc.cc:205] Tried to move back new op lower bound from 7332594157444329472 to 7332594157397192704. Current Snapshot: MvccSnapshot[applied={T|T < 7332594157444329472}]
I20260923 17:22:00.498656 5578 mvcc.cc:205] Tried to move back new op lower bound from 7332594157444329472 to 7332594157397192704. Current Snapshot: MvccSnapshot[applied={T|T < 7332594157444329472}]
I20260923 17:22:00.501703 5523 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bda6f2e84a1d418bb06d503182a741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.501845 5523 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:00.503242 5538 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bda6f2e84a1d418bb06d503182a741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda6f2e84a1d418bb06d503182a741ff" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 46189 } } peers { permanent_uuid: "5ec440819ceb4908bfba3064433ef9b3" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 34707 } } peers { permanent_uuid: "e4525c5c20ad44d7b67c66fc3b740c99" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 35723 } } }
I20260923 17:22:00.503374 5538 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:22:00.510703 5576 mvcc.cc:205] Tried to move back new op lower bound from 7332594157444329472 to 7332594157397192704. Current Snapshot: MvccSnapshot[applied={T|T < 7332594157444329472}]
I20260923 17:22:00.511008 5574 catalog_manager.cc:1347] Generated new cluster ID: a24cc2acf5c044cea06224d664378719
I20260923 17:22:00.511072 5574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:00.546317 5574 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:00.548022 5574 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:00.572631 5574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda6f2e84a1d418bb06d503182a741ff: Generated new TSK 0
I20260923 17:22:00.574448 5574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:22:01.373708 5554 catalog_manager.cc:1259] Loaded cluster ID: a24cc2acf5c044cea06224d664378719
I20260923 17:22:01.373826 5554 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: loading cluster ID for follower catalog manager: success
I20260923 17:22:01.375478 5554 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: acquiring CA information for follower catalog manager: success
I20260923 17:22:01.376731 5554 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ec440819ceb4908bfba3064433ef9b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:22:01.382194 5570 catalog_manager.cc:1259] Loaded cluster ID: a24cc2acf5c044cea06224d664378719
I20260923 17:22:01.382264 5570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: loading cluster ID for follower catalog manager: success
I20260923 17:22:01.384054 5570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: acquiring CA information for follower catalog manager: success
I20260923 17:22:01.385224 5570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4525c5c20ad44d7b67c66fc3b740c99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:22:01.633330 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5294
I20260923 17:22:01.639856 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5369
I20260923 17:22:01.645813 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5446
2026-09-23T17:22:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3168 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:01 dist-test-slave-9q7f krb5kdc[5588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:01 dist-test-slave-9q7f krb5kdc[5588](info): set up 2 sockets
Sep 23 17:22:01 dist-test-slave-9q7f krb5kdc[5588](info): commencing operation
krb5kdc: starting...
W20260923 17:22:02.680289 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:22:02 dist-test-slave-9q7f krb5kdc[5588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:22:02.805223 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:33077
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:33077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:02.917390 5601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:02.917677 5601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:02.917718 5601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:02.921723 5601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:02.921787 5601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:02.921806 5601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:02.921823 5601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:02.921838 5601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:02.927125 5601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:33077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:33077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:02.928120 5601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:02.929059 5601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:02.937920 5606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:02.937971 5607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:02.937923 5609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:02.938210 5601 server_base.cc:1034] running on GCE node
I20260923 17:22:02.938906 5601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:02.940034 5601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:02.941215 5601 hybrid_clock.cc:648] HybridClock initialized: now 1790184122941192 us; error 36 us; skew 500 ppm
Sep 23 17:22:02 dist-test-slave-9q7f krb5kdc[5588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184122, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:02.944097 5601 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:22:02.945014 5601 webserver.cc:460] Webserver started at http://127.23.214.190:36813/ using document root <none> and password file <none>
I20260923 17:22:02.945569 5601 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:02.945636 5601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:02.945850 5601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:02.947907 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "0af7f884c36f434ea570e15471c86790"
format_stamp: "Formatted at 2026-09-23 17:22:02 on dist-test-slave-9q7f"
server_key: "834a8723a87c3f00f77bd08ac2b7365c"
server_key_iv: "4b16eb0c6e6159d2141b5e1b9e578e55"
server_key_version: "encryptionkey@0"
I20260923 17:22:02.948525 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "0af7f884c36f434ea570e15471c86790"
format_stamp: "Formatted at 2026-09-23 17:22:02 on dist-test-slave-9q7f"
server_key: "834a8723a87c3f00f77bd08ac2b7365c"
server_key_iv: "4b16eb0c6e6159d2141b5e1b9e578e55"
server_key_version: "encryptionkey@0"
I20260923 17:22:02.952478 5601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 17:22:02.955084 5616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:02.956358 5601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:02.956516 5601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "0af7f884c36f434ea570e15471c86790"
format_stamp: "Formatted at 2026-09-23 17:22:02 on dist-test-slave-9q7f"
server_key: "834a8723a87c3f00f77bd08ac2b7365c"
server_key_iv: "4b16eb0c6e6159d2141b5e1b9e578e55"
server_key_version: "encryptionkey@0"
I20260923 17:22:02.956665 5601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:02.973057 5601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:02.976475 5601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:02.976698 5601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:02.983743 5601 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:33077
I20260923 17:22:02.983789 5668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:33077 every 8 connection(s)
I20260923 17:22:02.984710 5601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:02.987471 5669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:02.991784 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5601
I20260923 17:22:02.991926 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:22:02.992192 24410 external_mini_cluster.cc:1428] Setting key a960ad098256152add51faa0e89d1c76
I20260923 17:22:02.993419 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790: Bootstrap starting.
I20260923 17:22:02.995653 5669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:02.996400 5669 log.cc:824] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:02.998288 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790: No bootstrap required, opened a new log
Sep 23 17:22:02 dist-test-slave-9q7f krb5kdc[5588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:03.002337 5669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7f884c36f434ea570e15471c86790" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33077 } }
I20260923 17:22:03.002558 5669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:03.002647 5669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af7f884c36f434ea570e15471c86790, State: Initialized, Role: FOLLOWER
I20260923 17:22:03.003125 5669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7f884c36f434ea570e15471c86790" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33077 } }
I20260923 17:22:03.003254 5669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:03.003336 5669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:03.003449 5669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:03.004325 5669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7f884c36f434ea570e15471c86790" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33077 } }
I20260923 17:22:03.004667 5669 leader_election.cc:302] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af7f884c36f434ea570e15471c86790; no voters:
I20260923 17:22:03.005016 5669 leader_election.cc:288] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:03.005087 5674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:03.005271 5674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [term 1 LEADER]: Becoming Leader. State: Replica: 0af7f884c36f434ea570e15471c86790, State: Running, Role: LEADER
I20260923 17:22:03.005630 5674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7f884c36f434ea570e15471c86790" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33077 } }
I20260923 17:22:03.006361 5669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:03.007452 5675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af7f884c36f434ea570e15471c86790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7f884c36f434ea570e15471c86790" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33077 } } }
I20260923 17:22:03.007493 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af7f884c36f434ea570e15471c86790. Latest consensus state: current_term: 1 leader_uuid: "0af7f884c36f434ea570e15471c86790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7f884c36f434ea570e15471c86790" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33077 } } }
I20260923 17:22:03.007573 5675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:03.007601 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:03.007665 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:02.993538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.23.214.190:33077)
0923 17:22:02.994018 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:02.994033 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:02.994070 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:22:02.994706 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:02.994722 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:02.995035 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:02.995375 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:02.996188 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:02.997231 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:02.998155 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:02.998683 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:02.999018 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:03.001959 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:03.001964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:03.001978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:03.002005 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:03.003946 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:03.004381 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:03.004387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:03.004394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:03.004531 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:03.004811 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:03.004814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:03.004815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:03.005369 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:03.006201 (+ 832us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:03.006318 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 17:22:03.006609 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":133,"threads_started":1}
I20260923 17:22:03.007973 5683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:03.010836 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:22:03.011256 5683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:03.017239 5683 catalog_manager.cc:1347] Generated new cluster ID: 3d51f4a196ca488a85e4e75c35783450
I20260923 17:22:03.017318 5683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:03.036909 5683 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:03.038506 5683 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:03.042577 5683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af7f884c36f434ea570e15471c86790: Generated new TSK 0
I20260923 17:22:03.043315 5683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:22:03.066418 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5601
2026-09-23T17:22:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:03 dist-test-slave-9q7f krb5kdc[5696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:03 dist-test-slave-9q7f krb5kdc[5696](info): set up 2 sockets
Sep 23 17:22:03 dist-test-slave-9q7f krb5kdc[5696](info): commencing operation
krb5kdc: starting...
W20260923 17:22:04.188230 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:22:04 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:22:04.312295 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42571
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:04.421595 5710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:04.421980 5710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:04.422084 5710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:04.426153 5710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:04.426255 5710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:04.426295 5710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:04.426334 5710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:04.426366 5710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:04.431795 5710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:04.432974 5710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:04.434072 5710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:04.443318 5715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.443338 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.443334 5718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:04.443603 5710 server_base.cc:1034] running on GCE node
I20260923 17:22:04.444267 5710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:04.445446 5710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:04.446650 5710 hybrid_clock.cc:648] HybridClock initialized: now 1790184124446635 us; error 27 us; skew 500 ppm
Sep 23 17:22:04 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:04.449461 5710 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:22:04.450534 5710 webserver.cc:460] Webserver started at http://127.23.214.190:34751/ using document root <none> and password file <none>
I20260923 17:22:04.451086 5710 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:04.451151 5710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:04.451318 5710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:04.453420 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "9cc7d3320ff3462685c6ffdab493e27f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "e58b6e0f0a9563f3c6a954907defb924"
server_key_iv: "ceb7f72eb6b1d49bec9690d4a57e6bae"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.454041 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "9cc7d3320ff3462685c6ffdab493e27f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "e58b6e0f0a9563f3c6a954907defb924"
server_key_iv: "ceb7f72eb6b1d49bec9690d4a57e6bae"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.457940 5710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:22:04.460438 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.461725 5710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:04.461828 5710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "9cc7d3320ff3462685c6ffdab493e27f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "e58b6e0f0a9563f3c6a954907defb924"
server_key_iv: "ceb7f72eb6b1d49bec9690d4a57e6bae"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.461987 5710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:04.487694 5710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:04.490669 5710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:04.490828 5710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:04.497975 5710 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:42571
I20260923 17:22:04.498266 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:42571 every 8 connection(s)
I20260923 17:22:04.498972 5710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:04.501710 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:04.507264 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Bootstrap starting.
I20260923 17:22:04.508880 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5710
I20260923 17:22:04.509008 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:22:04.509312 24410 external_mini_cluster.cc:1428] Setting key cfa1442520bf49d9ec837eba57c5930e
I20260923 17:22:04.510134 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:04.511059 5780 log.cc:824] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:04.512982 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: No bootstrap required, opened a new log
I20260923 17:22:04.516460 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:04.516705 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:04.516793 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc7d3320ff3462685c6ffdab493e27f, State: Initialized, Role: FOLLOWER
I20260923 17:22:04.517258 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:04.517385 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:04.517470 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:04.517588 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:04.518661 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
Sep 23 17:22:04 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:04.519044 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc7d3320ff3462685c6ffdab493e27f; no voters:
I20260923 17:22:04.519421 5780 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:04.519486 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:04.519742 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 LEADER]: Becoming Leader. State: Replica: 9cc7d3320ff3462685c6ffdab493e27f, State: Running, Role: LEADER
I20260923 17:22:04.520388 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:04.520851 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:04.522435 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc7d3320ff3462685c6ffdab493e27f. Latest consensus state: current_term: 1 leader_uuid: "9cc7d3320ff3462685c6ffdab493e27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } } }
I20260923 17:22:04.522595 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:04.523038 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc7d3320ff3462685c6ffdab493e27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } } }
I20260923 17:22:04.523156 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:04.525415 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.510627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.23.214.190:42571)
0923 17:22:04.511103 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:04.511120 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:04.511157 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:22:04.512110 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:04.512131 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:04.512475 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:04.512844 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:04.513697 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.514819 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.515749 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.516269 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.516454 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.519626 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:04.519633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:04.519653 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:04.519687 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:04.521563 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.522239 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.522247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.522256 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.522420 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.522778 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.522781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.522783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.523451 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:04.524313 (+ 862us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:04.524453 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 17:22:04.524689 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
W20260923 17:22:04.525435 5800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:22:04.525552 5800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:22:04.525688 5799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:04.526633 5799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:04.532320 5799 catalog_manager.cc:1347] Generated new cluster ID: 8a976f0a29e34d57bce1c0b16f2d080c
I20260923 17:22:04.532400 5799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:04.547794 5799 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:04.549122 5799 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:04.555850 5799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Generated new TSK 0
I20260923 17:22:04.556703 5799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:22:04.589767 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:42571
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:04.704973 5806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:04.705276 5806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:04.705344 5806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:04.709348 5806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:04.709436 5806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:04.709523 5806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:22:04.714857 5806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:04.715922 5806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:04.716965 5806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:04.726683 5811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:04.726733 5806 server_base.cc:1034] running on GCE node
W20260923 17:22:04.726677 5812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.726677 5814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:04.727304 5806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:04.727927 5806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:04.729094 5806 hybrid_clock.cc:648] HybridClock initialized: now 1790184124729059 us; error 44 us; skew 500 ppm
Sep 23 17:22:04 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:04.740124 5806 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:22:04.741130 5806 webserver.cc:460] Webserver started at http://127.23.214.129:38611/ using document root <none> and password file <none>
I20260923 17:22:04.741710 5806 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:04.741799 5806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:04.742091 5806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:04.744303 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "b172d24899374956bf3a9c3dea570f8f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "bfb434c2e33c385455e06fdb34536911"
server_key_iv: "de621cce6a5ac91b77a27673e3de4738"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.744825 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "b172d24899374956bf3a9c3dea570f8f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "bfb434c2e33c385455e06fdb34536911"
server_key_iv: "de621cce6a5ac91b77a27673e3de4738"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.748536 5806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 17:22:04.751073 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.752341 5806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:04.752460 5806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b172d24899374956bf3a9c3dea570f8f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "bfb434c2e33c385455e06fdb34536911"
server_key_iv: "de621cce6a5ac91b77a27673e3de4738"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.752576 5806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:04.766496 5806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:04.769658 5806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:04.769856 5806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:04.770512 5806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:04.771313 5806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:04.771358 5806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.771420 5806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:04.771451 5806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.780558 5806 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:45045
I20260923 17:22:04.780587 5935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:45045 every 8 connection(s)
I20260923 17:22:04.781432 5806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:04.786731 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5806
I20260923 17:22:04.786862 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:04.787182 24410 external_mini_cluster.cc:1428] Setting key 959e1ee8c916127e7fca45f11e79433b
Sep 23 17:22:04 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:04.796085 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.783213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:49593 (local address 127.23.214.190:42571)
0923 17:22:04.783346 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:04.783350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:04.784162 (+ 812us) server_negotiation.cc:407] Connection header received
0923 17:22:04.785578 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:04.785582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:04.785644 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:04.785739 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:04.786999 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.787750 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.789092 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.789315 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.789391 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.792237 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:04.792244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:04.792250 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:04.792283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:04.794074 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.794532 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.794536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.794547 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.794650 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:04.794958 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:04.794961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:04.794965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:04.795206 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:04.795312 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:04.795765 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 17:22:04.795892 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:22:04.796769 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.783581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:49593)
0923 17:22:04.784010 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:04.784066 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:04.785334 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:04.785953 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:04.785960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:04.786313 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:04.786802 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.786824 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.787878 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.787881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.788951 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.788965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.789458 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.789465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.789705 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.790422 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:04.790438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:04.792078 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:04.794186 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:04.794191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:04.794202 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:04.794435 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:04.794743 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:04.794745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:04.794747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:04.794866 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:04.795302 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:04.795306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:04.795514 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 17:22:04.795681 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:22:04.795912 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:22:04.798060 5936 heartbeater.cc:347] Connected to a master server at 127.23.214.190:42571
I20260923 17:22:04.798292 5936 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:04.798771 5936 heartbeater.cc:511] Master 127.23.214.190:42571 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:22:04.800251 5743 ts_manager.cc:194] Registered new tserver with Master: b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045)
I20260923 17:22:04.801628 5743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:49593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:22:04.814710 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:42571
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:04.925619 5941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:04.925868 5941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:04.925989 5941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:04.930022 5941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:04.930083 5941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:04.930192 5941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:22:04.935392 5941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:04.936435 5941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:04.937454 5941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:04.947150 5946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.947150 5949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:04.947250 5947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:04.947403 5941 server_base.cc:1034] running on GCE node
I20260923 17:22:04.947875 5941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:04.948495 5941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:04.949653 5941 hybrid_clock.cc:648] HybridClock initialized: now 1790184124949618 us; error 46 us; skew 500 ppm
Sep 23 17:22:04 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:04.952361 5941 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:22:04.953204 5941 webserver.cc:460] Webserver started at http://127.23.214.130:33945/ using document root <none> and password file <none>
I20260923 17:22:04.953758 5941 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:04.953820 5941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:04.954056 5941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:04.956205 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "00da9567cbe6462594e5d66d9f5ab2dc"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "dd709e3494e4ad76c566111578472cd1"
server_key_iv: "dad7f739f75e43d4a9e04dc24fcf905f"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.956831 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "00da9567cbe6462594e5d66d9f5ab2dc"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "dd709e3494e4ad76c566111578472cd1"
server_key_iv: "dad7f739f75e43d4a9e04dc24fcf905f"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.960683 5941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:22:04.963131 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.964426 5941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:04.964555 5941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "00da9567cbe6462594e5d66d9f5ab2dc"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "dd709e3494e4ad76c566111578472cd1"
server_key_iv: "dad7f739f75e43d4a9e04dc24fcf905f"
server_key_version: "encryptionkey@0"
I20260923 17:22:04.964684 5941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:04.974264 5941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:04.977029 5941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:04.977224 5941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:04.977783 5941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:04.978614 5941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:04.978680 5941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.978746 5941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:04.978792 5941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:04.988081 5941 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:41397
I20260923 17:22:04.988152 6072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:41397 every 8 connection(s)
I20260923 17:22:04.989030 5941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:04.990739 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 5941
I20260923 17:22:04.990865 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:04.991171 24410 external_mini_cluster.cc:1428] Setting key f75ab41ebece875cef4c3b3f526d06fb
Sep 23 17:22:04 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184124, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:22:05.004666 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.991110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:57813 (local address 127.23.214.190:42571)
0923 17:22:04.991277 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:04.991283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:04.992187 (+ 904us) server_negotiation.cc:407] Connection header received
0923 17:22:04.994052 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:04.994055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:04.994106 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:04.994191 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:04.995564 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.996549 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.997772 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.998271 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.998449 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.000801 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.000806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.000811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.000849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.002674 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.003112 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.003115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.003117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.003240 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.003556 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.003560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.003561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.003749 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.003837 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.004371 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.004531 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
I20260923 17:22:05.005496 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:04.991560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:57813)
0923 17:22:04.992029 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:04.992082 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:04.993779 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:04.994385 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:04.994396 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:04.994894 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:04.995358 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.995372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.996679 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.996682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.997632 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:04.997641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:04.998444 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:04.998446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:04.998596 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:04.999192 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:04.999208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:05.000624 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:05.002775 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.002780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.002791 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.003021 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.003318 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.003321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.003322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.003448 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.003841 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:05.003846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:05.004094 (+ 248us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.004282 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.004526 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":135,"threads_started":1}
I20260923 17:22:05.006799 6073 heartbeater.cc:347] Connected to a master server at 127.23.214.190:42571
I20260923 17:22:05.007067 6073 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:05.007670 6073 heartbeater.cc:511] Master 127.23.214.190:42571 requested a full tablet report, sending...
I20260923 17:22:05.008754 5743 ts_manager.cc:194] Registered new tserver with Master: 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397)
I20260923 17:22:05.009341 5743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:57813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:22:05.020290 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:42571
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:05.131678 6079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:05.131985 6079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:05.132066 6079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:05.136090 6079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:05.136185 6079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:05.136279 6079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:22:05.141737 6079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:05.143144 6079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:05.144690 6079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:05.155005 6084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.155001 6085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.155025 6087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:05.155421 6079 server_base.cc:1034] running on GCE node
I20260923 17:22:05.155957 6079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:05.156705 6079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:05.157940 6079 hybrid_clock.cc:648] HybridClock initialized: now 1790184125157912 us; error 58 us; skew 500 ppm
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.161036 6079 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:22:05.162009 6079 webserver.cc:460] Webserver started at http://127.23.214.131:44669/ using document root <none> and password file <none>
I20260923 17:22:05.162629 6079 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:05.162729 6079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:05.162982 6079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:05.165272 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "de85856704c0469e95f6a86bbd7d6719"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-9q7f"
server_key: "2f29f0f9981cdef3506b14eb0f8ebd77"
server_key_iv: "7d1e22e60c623f123d80649996dc11d4"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.165969 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "de85856704c0469e95f6a86bbd7d6719"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-9q7f"
server_key: "2f29f0f9981cdef3506b14eb0f8ebd77"
server_key_iv: "7d1e22e60c623f123d80649996dc11d4"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.170143 6079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260923 17:22:05.172633 6094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.173964 6079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260923 17:22:05.174111 6079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "de85856704c0469e95f6a86bbd7d6719"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-9q7f"
server_key: "2f29f0f9981cdef3506b14eb0f8ebd77"
server_key_iv: "7d1e22e60c623f123d80649996dc11d4"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.174230 6079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:05.199688 6079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:05.202924 6079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:05.203156 6079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:05.203804 6079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:05.204711 6079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:05.204784 6079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.204860 6079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:05.204905 6079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.214540 6079 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:34545
I20260923 17:22:05.214561 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:34545 every 8 connection(s)
I20260923 17:22:05.215552 6079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:05.216854 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 6079
I20260923 17:22:05.216982 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:05.217299 24410 external_mini_cluster.cc:1428] Setting key 0503dad3b236f4d97a413ec125a4975d
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:05.230383 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.218014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:35131 (local address 127.23.214.190:42571)
0923 17:22:05.218169 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.218174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.218731 (+ 557us) server_negotiation.cc:407] Connection header received
0923 17:22:05.220317 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.220320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.220366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.220459 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.221951 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.222589 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.223781 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.223994 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.224086 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.226821 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.226824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.226827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.226854 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.228324 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.228809 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.228812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.228813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.228888 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.229230 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.229232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.229233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.229400 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.229475 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.230044 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.230177 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:22:05.231360 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.217994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:35131)
0923 17:22:05.218552 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.218621 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.220121 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.220702 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.220717 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.221096 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:05.221767 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.221785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.222712 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.222715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.223665 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.223674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.224126 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.224129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.224283 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.224888 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:05.224906 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:05.226650 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:05.228428 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.228433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.228445 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.228723 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.228997 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.229000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.229001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.229146 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.229488 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:05.229494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:05.229752 (+ 258us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.229996 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.230361 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":175,"threads_started":1}
I20260923 17:22:05.232370 6209 heartbeater.cc:347] Connected to a master server at 127.23.214.190:42571
I20260923 17:22:05.232625 6209 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:05.233127 6209 heartbeater.cc:511] Master 127.23.214.190:42571 requested a full tablet report, sending...
I20260923 17:22:05.234119 5743 ts_manager.cc:194] Registered new tserver with Master: de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545)
I20260923 17:22:05.234644 5743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:35131
I20260923 17:22:05.242806 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:22:05.251717 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.244823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.23.214.190:42571)
0923 17:22:05.244944 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.244948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.245115 (+ 167us) server_negotiation.cc:407] Connection header received
0923 17:22:05.245217 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.245219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.245254 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.245318 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.245991 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.246485 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.247261 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.247460 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.247530 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.248792 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.248795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.248797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.248819 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.250283 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.250645 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.250649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.250650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.250745 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.250969 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.250972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.250973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.251173 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.251274 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.251494 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.251598 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:22:05.254454 5743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:05.257679 5743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:05.271481 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.265861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:41397 (local address 127.0.0.1:50818)
0923 17:22:05.266380 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.266425 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.266570 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.267455 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.267464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.267492 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:05.267624 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.267631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.268856 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.268862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.270071 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.270081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.270918 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.270921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.271034 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.271084 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.271209 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.271326 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":86,"threads_started":1}
I20260923 17:22:05.271486 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.266007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.23.214.130:41397)
0923 17:22:05.266429 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.266440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.266734 (+ 294us) server_negotiation.cc:407] Connection header received
0923 17:22:05.266820 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.266827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.267081 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.267245 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:05.267730 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.268743 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.270199 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.270793 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.270929 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.271065 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.271239 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.271326 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":126,"threads_started":1}
I20260923 17:22:05.272439 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.266014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45045 (local address 127.0.0.1:38678)
0923 17:22:05.267368 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.267383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.267488 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.268029 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.268032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.268059 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:05.268152 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.268158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.269630 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.269633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.270893 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.270903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.272029 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.272035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.272136 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.272159 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.272246 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.272315 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":105,"threads_started":1}
I20260923 17:22:05.272538 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.266797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.23.214.129:45045)
0923 17:22:05.267245 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.267261 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.267468 (+ 207us) server_negotiation.cc:407] Connection header received
0923 17:22:05.267669 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.267676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.267879 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.268031 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:05.268251 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.269475 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.271048 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.271862 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.272043 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.272176 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.272274 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.272368 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":112,"threads_started":1}
I20260923 17:22:05.272650 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.266882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34545 (local address 127.0.0.1:44166)
0923 17:22:05.267740 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.267752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.267823 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.268480 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.268483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.268500 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:05.268571 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.268576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.269948 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.269951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.271063 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.271071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.272242 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.272246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.272331 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.272352 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.272422 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.272489 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":103,"threads_started":1}
I20260923 17:22:05.272985 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.267310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.23.214.131:34545)
0923 17:22:05.268054 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.268067 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.268090 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:22:05.268163 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.268171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.268346 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.268499 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:05.268710 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.269789 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.271196 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.271974 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.272073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.272176 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.272742 (+ 566us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.272848 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":141,"threads_started":1}
I20260923 17:22:05.275122 5870 tablet_service.cc:1467] Processing CreateTablet for tablet 92edd5fb0cd54e8d80f6bd922b4d416e (DEFAULT_TABLE table=test-table [id=0e90d3fe406748b39751a7220bf0db05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:05.275153 6005 tablet_service.cc:1467] Processing CreateTablet for tablet 92edd5fb0cd54e8d80f6bd922b4d416e (DEFAULT_TABLE table=test-table [id=0e90d3fe406748b39751a7220bf0db05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:05.275529 6142 tablet_service.cc:1467] Processing CreateTablet for tablet 92edd5fb0cd54e8d80f6bd922b4d416e (DEFAULT_TABLE table=test-table [id=0e90d3fe406748b39751a7220bf0db05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:05.276095 5870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92edd5fb0cd54e8d80f6bd922b4d416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:05.276182 6005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92edd5fb0cd54e8d80f6bd922b4d416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:05.276386 6142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92edd5fb0cd54e8d80f6bd922b4d416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:05.282779 6226 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Bootstrap starting.
I20260923 17:22:05.283954 6227 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Bootstrap starting.
I20260923 17:22:05.283954 6228 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Bootstrap starting.
I20260923 17:22:05.285023 6226 tablet_bootstrap.cc:654] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:05.286155 6227 tablet_bootstrap.cc:654] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:05.286185 6226 log.cc:824] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:05.286661 6228 tablet_bootstrap.cc:654] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:05.287456 6227 log.cc:824] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:05.287456 6228 log.cc:824] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:05.288321 6226 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: No bootstrap required, opened a new log
I20260923 17:22:05.288524 6226 ts_tablet_manager.cc:1397] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:22:05.289215 6228 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: No bootstrap required, opened a new log
I20260923 17:22:05.289391 6227 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: No bootstrap required, opened a new log
I20260923 17:22:05.289413 6228 ts_tablet_manager.cc:1397] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:22:05.289577 6227 ts_tablet_manager.cc:1397] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:22:05.292243 6228 raft_consensus.cc:348] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.292481 6228 raft_consensus.cc:374] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:05.292567 6228 raft_consensus.cc:729] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00da9567cbe6462594e5d66d9f5ab2dc, State: Initialized, Role: FOLLOWER
I20260923 17:22:05.292430 6226 raft_consensus.cc:348] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.292721 6226 raft_consensus.cc:374] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:05.292783 6226 raft_consensus.cc:729] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b172d24899374956bf3a9c3dea570f8f, State: Initialized, Role: FOLLOWER
I20260923 17:22:05.293025 6228 consensus_queue.cc:260] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.293196 6226 consensus_queue.cc:260] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.293288 6227 raft_consensus.cc:348] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.293545 6227 raft_consensus.cc:374] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:05.293639 6227 raft_consensus.cc:729] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de85856704c0469e95f6a86bbd7d6719, State: Initialized, Role: FOLLOWER
I20260923 17:22:05.293710 6073 heartbeater.cc:503] Master 127.23.214.190:42571 was elected leader, sending a full tablet report...
I20260923 17:22:05.294113 6228 ts_tablet_manager.cc:1428] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:22:05.294263 6227 consensus_queue.cc:260] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.294971 6209 heartbeater.cc:503] Master 127.23.214.190:42571 was elected leader, sending a full tablet report...
I20260923 17:22:05.295274 6227 ts_tablet_manager.cc:1428] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:22:05.296808 5936 heartbeater.cc:503] Master 127.23.214.190:42571 was elected leader, sending a full tablet report...
I20260923 17:22:05.297070 6226 ts_tablet_manager.cc:1428] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:22:05.319658 6232 raft_consensus.cc:482] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:05.319852 6232 raft_consensus.cc:504] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.321046 6232 leader_election.cc:288] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045), de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545)
I20260923 17:22:05.324723 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.321200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45045 (local address 127.23.214.130:49265)
0923 17:22:05.321367 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.321385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.321492 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.321761 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.321779 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.321802 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:05.321977 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.321984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.322861 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.322867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.323654 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.323663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.324298 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.324302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.324406 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.324427 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.324516 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.324591 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 17:22:05.324723 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.321331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:49265 (local address 127.23.214.129:45045)
0923 17:22:05.321492 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.321497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.321514 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:22:05.321577 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.321580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.321658 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.321743 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:05.322107 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.322756 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.323785 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.324175 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.324299 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.324351 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.324533 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.324609 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:22:05.324981 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.321503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:54773 (local address 127.23.214.131:34545)
0923 17:22:05.321654 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.321658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.321819 (+ 161us) server_negotiation.cc:407] Connection header received
0923 17:22:05.322050 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.322054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.322109 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.322192 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:05.322437 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.322952 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.324063 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.324462 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.324588 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.324635 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.324806 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.324877 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:22:05.324995 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.321297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34545 (local address 127.23.214.130:54773)
0923 17:22:05.321740 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.321754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.321943 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.322202 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.322205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.322220 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:05.322337 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.322342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.323058 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.323062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.323948 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.323957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.324588 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.324592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.324692 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.324709 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.324790 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.324854 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260923 17:22:05.325475 5890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92edd5fb0cd54e8d80f6bd922b4d416e" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b172d24899374956bf3a9c3dea570f8f" is_pre_election: true
I20260923 17:22:05.325695 6162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92edd5fb0cd54e8d80f6bd922b4d416e" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de85856704c0469e95f6a86bbd7d6719" is_pre_election: true
I20260923 17:22:05.325819 5890 raft_consensus.cc:2454] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 0.
I20260923 17:22:05.325994 6162 raft_consensus.cc:2454] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 0.
I20260923 17:22:05.326324 5957 leader_election.cc:302] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00da9567cbe6462594e5d66d9f5ab2dc, b172d24899374956bf3a9c3dea570f8f; no voters:
I20260923 17:22:05.326608 6232 raft_consensus.cc:2781] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:05.326692 6232 raft_consensus.cc:482] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:05.326741 6232 raft_consensus.cc:3037] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:05.327772 6232 raft_consensus.cc:504] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.328133 6232 leader_election.cc:288] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 election: Requested vote from peers b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045), de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545)
I20260923 17:22:05.328449 5890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92edd5fb0cd54e8d80f6bd922b4d416e" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b172d24899374956bf3a9c3dea570f8f"
I20260923 17:22:05.328449 6162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92edd5fb0cd54e8d80f6bd922b4d416e" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de85856704c0469e95f6a86bbd7d6719"
I20260923 17:22:05.328574 5890 raft_consensus.cc:3037] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:05.328572 6162 raft_consensus.cc:3037] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:05.329840 6162 raft_consensus.cc:2454] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 1.
I20260923 17:22:05.329840 5890 raft_consensus.cc:2454] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 1.
I20260923 17:22:05.330143 5958 leader_election.cc:302] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00da9567cbe6462594e5d66d9f5ab2dc, de85856704c0469e95f6a86bbd7d6719; no voters:
I20260923 17:22:05.330332 6232 raft_consensus.cc:2781] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:05.330549 6232 raft_consensus.cc:686] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 LEADER]: Becoming Leader. State: Replica: 00da9567cbe6462594e5d66d9f5ab2dc, State: Running, Role: LEADER
I20260923 17:22:05.330873 6232 consensus_queue.cc:237] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.334301 5742 catalog_manager.cc:5495] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130). New cstate: current_term: 1 leader_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } health_report { overall_health: UNKNOWN } } }
I20260923 17:22:05.349601 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.346085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.23.214.130:41397)
0923 17:22:05.346216 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.346223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.346238 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:22:05.346425 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.346428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.346527 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.346636 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:05.346871 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.347376 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.348206 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.348416 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.348526 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.348808 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:05.349198 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:05.349328 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.349376 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.349435 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:22:05.355340 6162 raft_consensus.cc:1261] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Refusing update from remote peer 00da9567cbe6462594e5d66d9f5ab2dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:05.355340 5890 raft_consensus.cc:1261] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Refusing update from remote peer 00da9567cbe6462594e5d66d9f5ab2dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:05.356124 6236 consensus_queue.cc:1035] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:05.356346 6232 consensus_queue.cc:1035] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:05.363400 6239 mvcc.cc:205] Tried to move back new op lower bound from 7332594177448165376 to 7332594177357762560. Current Snapshot: MvccSnapshot[applied={T|T < 7332594177448165376}]
I20260923 17:22:05.364113 6241 mvcc.cc:205] Tried to move back new op lower bound from 7332594177448165376 to 7332594177357762560. Current Snapshot: MvccSnapshot[applied={T|T < 7332594177448165376}]
I20260923 17:22:05.367416 6240 mvcc.cc:205] Tried to move back new op lower bound from 7332594177448165376 to 7332594177357762560. Current Snapshot: MvccSnapshot[applied={T|T < 7332594177448165376}]
I20260923 17:22:05.372717 5742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:05.372862 5742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:05.375288 5742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Sending DeleteTablet for 3 replicas of tablet 92edd5fb0cd54e8d80f6bd922b4d416e
I20260923 17:22:05.376011 5870 tablet_service.cc:1514] Processing DeleteTablet for tablet 92edd5fb0cd54e8d80f6bd922b4d416e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:05 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:38678
I20260923 17:22:05.376190 6142 tablet_service.cc:1514] Processing DeleteTablet for tablet 92edd5fb0cd54e8d80f6bd922b4d416e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:05 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:44166
I20260923 17:22:05.376171 6005 tablet_service.cc:1514] Processing DeleteTablet for tablet 92edd5fb0cd54e8d80f6bd922b4d416e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:05 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:50818
I20260923 17:22:05.376653 6249 tablet_replica.cc:330] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: stopping tablet replica
I20260923 17:22:05.376857 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5806
I20260923 17:22:05.377477 6249 raft_consensus.cc:2229] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:05.377820 6249 raft_consensus.cc:2258] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:05.378206 6250 tablet_replica.cc:330] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: stopping tablet replica
I20260923 17:22:05.378580 6250 raft_consensus.cc:2229] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:05.378892 6250 raft_consensus.cc:2258] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:05.379616 6249 ts_tablet_manager.cc:1905] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:05.380358 6250 ts_tablet_manager.cc:1905] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:05.383054 6250 ts_tablet_manager.cc:1918] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:05.383157 6250 log.cc:1196] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/92edd5fb0cd54e8d80f6bd922b4d416e
I20260923 17:22:05.383394 6249 ts_tablet_manager.cc:1918] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:05.383492 6250 ts_tablet_manager.cc:1939] T 92edd5fb0cd54e8d80f6bd922b4d416e P de85856704c0469e95f6a86bbd7d6719: Deleting consensus metadata
I20260923 17:22:05.383597 6249 log.cc:1196] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/92edd5fb0cd54e8d80f6bd922b4d416e
I20260923 17:22:05.383972 6249 ts_tablet_manager.cc:1939] T 92edd5fb0cd54e8d80f6bd922b4d416e P 00da9567cbe6462594e5d66d9f5ab2dc: Deleting consensus metadata
I20260923 17:22:05.384727 5727 catalog_manager.cc:4841] TS de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545): tablet 92edd5fb0cd54e8d80f6bd922b4d416e (table test-table [id=0e90d3fe406748b39751a7220bf0db05]) successfully deleted
W20260923 17:22:05.385058 5726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:45045 (error 108)
I20260923 17:22:05.385681 5728 catalog_manager.cc:4841] TS 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397): tablet 92edd5fb0cd54e8d80f6bd922b4d416e (table test-table [id=0e90d3fe406748b39751a7220bf0db05]) successfully deleted
W20260923 17:22:05.386161 6224 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:05.385839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45045 (local address 127.0.0.1:38694)
0923 17:22:05.385996 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.386066 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:45045: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 17:22:05.386312 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5941
W20260923 17:22:05.386318 5726 catalog_manager.cc:4584] TS b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92edd5fb0cd54e8d80f6bd922b4d416e: Network error: Client connection negotiation failed: client connection to 127.23.214.129:45045: connect: Connection reset by peer (error 104)
I20260923 17:22:05.392481 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 6079
I20260923 17:22:05.397840 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 5710
I20260923 17:22:05.404604 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42571
--webserver_interface=127.23.214.190
--webserver_port=34751
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:05.516424 6252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:05.516680 6252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:05.516721 6252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:05.520634 6252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:05.520684 6252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:05.520704 6252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:05.520720 6252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:05.520735 6252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:05.525740 6252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=34751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:05.526782 6252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:05.527796 6252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:05.541738 6261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.541838 6259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:05.541965 6252 server_base.cc:1034] running on GCE node
W20260923 17:22:05.541901 6258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:05.542695 6252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:05.543859 6252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:05.545058 6252 hybrid_clock.cc:648] HybridClock initialized: now 1790184125545041 us; error 37 us; skew 500 ppm
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.547904 6252 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:22:05.548820 6252 webserver.cc:460] Webserver started at http://127.23.214.190:34751/ using document root <none> and password file <none>
I20260923 17:22:05.549397 6252 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:05.549504 6252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:05.553383 6252 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:22:05.555526 6268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.556753 6252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:05.556890 6252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "9cc7d3320ff3462685c6ffdab493e27f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "e58b6e0f0a9563f3c6a954907defb924"
server_key_iv: "ceb7f72eb6b1d49bec9690d4a57e6bae"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.557500 6252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:05.579420 6252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:05.582659 6252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:05.582868 6252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:05.590332 6252 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:42571
I20260923 17:22:05.590655 6320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:42571 every 8 connection(s)
I20260923 17:22:05.591444 6252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:05.595734 6322 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:22:05.597333 6322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Bootstrap starting.
I20260923 17:22:05.601145 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 6252
I20260923 17:22:05.601589 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:45045
--local_ip_for_outbound_sockets=127.23.214.129
--tserver_master_addrs=127.23.214.190:42571
--webserver_port=38611
--webserver_interface=127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:05.615200 6322 log.cc:824] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:05.626075 6322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:05.626542 6322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Bootstrap complete.
I20260923 17:22:05.631222 6322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:05.631700 6322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc7d3320ff3462685c6ffdab493e27f, State: Initialized, Role: FOLLOWER
I20260923 17:22:05.632268 6322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:05.632447 6322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:05.632539 6322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:05.632675 6322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:22:05.635002 6322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:05.635383 6322 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc7d3320ff3462685c6ffdab493e27f; no voters:
I20260923 17:22:05.635780 6322 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:22:05.635890 6326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:22:05.636121 6326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 LEADER]: Becoming Leader. State: Replica: 9cc7d3320ff3462685c6ffdab493e27f, State: Running, Role: LEADER
I20260923 17:22:05.636461 6326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:05.637216 6322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:05.637552 6326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc7d3320ff3462685c6ffdab493e27f. Latest consensus state: current_term: 2 leader_uuid: "9cc7d3320ff3462685c6ffdab493e27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } } }
I20260923 17:22:05.637696 6326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:05.638083 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9cc7d3320ff3462685c6ffdab493e27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } } }
I20260923 17:22:05.638289 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:05.639492 6330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:05.645100 6330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:05.645859 6330 catalog_manager.cc:1259] Loaded cluster ID: 8a976f0a29e34d57bce1c0b16f2d080c
I20260923 17:22:05.645956 6330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:05.647626 6330 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 17:22:05.652596 6330 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 17:22:05.658447 6330 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:05.659246 6330 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Loaded TSK: 0
I20260923 17:22:05.660488 6330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:22:05.716944 6324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:05.717248 6324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:05.717332 6324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:05.721463 6324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:05.721560 6324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:05.721652 6324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:22:05.727078 6324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:45045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=38611
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:05.728170 6324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:05.729208 6324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:05.740302 6349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:05.740542 6324 server_base.cc:1034] running on GCE node
W20260923 17:22:05.740314 6350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.740429 6352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:05.741135 6324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:05.741829 6324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:05.743036 6324 hybrid_clock.cc:648] HybridClock initialized: now 1790184125743007 us; error 47 us; skew 500 ppm
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.746136 6324 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:22:05.747176 6324 webserver.cc:460] Webserver started at http://127.23.214.129:38611/ using document root <none> and password file <none>
I20260923 17:22:05.747752 6324 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:05.747854 6324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:05.751863 6324 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:22:05.754052 6359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.755362 6324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:22:05.755456 6324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b172d24899374956bf3a9c3dea570f8f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "bfb434c2e33c385455e06fdb34536911"
server_key_iv: "de621cce6a5ac91b77a27673e3de4738"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.756032 6324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:05.766016 6324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:05.769027 6324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:05.769239 6324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:05.769910 6324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:05.771003 6367 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:22:05.775072 6324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:22:05.775138 6324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260923 17:22:05.775185 6324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:22:05.777281 6324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:22:05.777338 6324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 17:22:05.777695 6367 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Bootstrap starting.
I20260923 17:22:05.788641 6324 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:45045
I20260923 17:22:05.790032 6324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:05.790894 6477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:45045 every 8 connection(s)
I20260923 17:22:05.798049 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 6324
I20260923 17:22:05.798535 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:41397
--local_ip_for_outbound_sockets=127.23.214.130
--tserver_master_addrs=127.23.214.190:42571
--webserver_port=33945
--webserver_interface=127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:05.799113 6367 log.cc:824] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:05.807337 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.792485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:55627 (local address 127.23.214.190:42571)
0923 17:22:05.792970 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.792991 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.793296 (+ 305us) server_negotiation.cc:407] Connection header received
0923 17:22:05.794470 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.794480 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.794786 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.795113 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.796221 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.797171 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.798702 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.799474 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.799672 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.802416 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.802420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.802432 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.802458 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.804153 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.804686 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.804691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.804695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.804803 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.805142 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.805146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.805148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.805478 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.805622 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.806203 (+ 581us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.806452 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":112,"threads_started":1}
I20260923 17:22:05.807350 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.792506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:55627)
0923 17:22:05.793134 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.793176 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.794065 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.795179 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.795186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.795591 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:05.796052 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.796067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.797334 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.797337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.798531 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.798544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.799622 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.799629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.799834 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.800542 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:05.800563 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:05.802223 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:05.804288 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.804293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.804306 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.804564 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.804899 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.804902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.804904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.805044 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.805606 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:05.805612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:05.805844 (+ 232us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.806077 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.806418 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":146,"threads_started":1}
I20260923 17:22:05.808593 6367 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:05.809057 6367 tablet_bootstrap.cc:492] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Bootstrap complete.
I20260923 17:22:05.809929 6478 heartbeater.cc:347] Connected to a master server at 127.23.214.190:42571
I20260923 17:22:05.810155 6478 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:05.810607 6367 ts_tablet_manager.cc:1397] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.011s
I20260923 17:22:05.810618 6478 heartbeater.cc:511] Master 127.23.214.190:42571 requested a full tablet report, sending...
I20260923 17:22:05.813300 6285 ts_manager.cc:194] Registered new tserver with Master: b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045)
I20260923 17:22:05.813546 6367 raft_consensus.cc:348] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.814097 6367 raft_consensus.cc:729] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b172d24899374956bf3a9c3dea570f8f, State: Initialized, Role: FOLLOWER
I20260923 17:22:05.814569 6367 consensus_queue.cc:260] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } } peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } }
I20260923 17:22:05.815534 6367 ts_tablet_manager.cc:1428] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:22:05.816845 6285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:55627
I20260923 17:22:05.819273 6478 heartbeater.cc:503] Master 127.23.214.190:42571 was elected leader, sending a full tablet report...
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:22:05.827589 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.816345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45045 (local address 127.0.0.1:38702)
0923 17:22:05.817152 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.817198 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.817317 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.818037 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.818043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.818459 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:05.818601 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.818609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.820016 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.820019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.820994 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.821010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.821749 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.821751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.821827 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.822455 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:05.822468 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:05.823881 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:05.825970 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.825978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.825983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.826233 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.826532 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.826535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.826536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.826611 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.827074 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:05.827079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:05.827179 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.827288 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.827444 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":120,"threads_started":1}
I20260923 17:22:05.827694 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.816433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.23.214.129:45045)
0923 17:22:05.817562 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.817572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.817604 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:22:05.817666 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.817671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.817893 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.818037 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.818732 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.819883 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.821107 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.821638 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.821742 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.824012 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.824021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.824033 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.824059 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.825841 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.826341 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.826347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.826349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.826441 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.826689 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.826692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.826694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.826957 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.827083 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.827382 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.827524 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"spinlock_wait_cycles":1280,"thread_start_us":100,"threads_started":1}
I20260923 17:22:05.828727 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 92edd5fb0cd54e8d80f6bd922b4d416e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:05 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:38702
I20260923 17:22:05.829694 6487 tablet_replica.cc:330] stopping tablet replica
I20260923 17:22:05.830044 6487 raft_consensus.cc:2229] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:05.830294 6487 raft_consensus.cc:2258] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:05.831264 6487 ts_tablet_manager.cc:1905] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:05.835071 6487 ts_tablet_manager.cc:1918] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:05.835183 6487 log.cc:1196] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/92edd5fb0cd54e8d80f6bd922b4d416e
I20260923 17:22:05.835443 6487 ts_tablet_manager.cc:1939] T 92edd5fb0cd54e8d80f6bd922b4d416e P b172d24899374956bf3a9c3dea570f8f: Deleting consensus metadata
I20260923 17:22:05.837191 6269 catalog_manager.cc:4841] TS b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045): tablet 92edd5fb0cd54e8d80f6bd922b4d416e (table test-table [id=0e90d3fe406748b39751a7220bf0db05]) successfully deleted
W20260923 17:22:05.918249 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:05.918561 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:05.918634 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:05.922645 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:05.922739 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:05.922832 6483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:22:05.928103 6483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:41397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=33945
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:05.929194 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:05.930254 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:05.939849 6495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.939868 6492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:05.939925 6493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:05.940233 6483 server_base.cc:1034] running on GCE node
I20260923 17:22:05.940652 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:05.941368 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:05.942521 6483 hybrid_clock.cc:648] HybridClock initialized: now 1790184125942502 us; error 30 us; skew 500 ppm
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:05.945521 6483 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:22:05.946485 6483 webserver.cc:460] Webserver started at http://127.23.214.130:33945/ using document root <none> and password file <none>
I20260923 17:22:05.947049 6483 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:05.947137 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:05.951648 6483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:22:05.953819 6502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.955178 6483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:22:05.955327 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "00da9567cbe6462594e5d66d9f5ab2dc"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "dd709e3494e4ad76c566111578472cd1"
server_key_iv: "dad7f739f75e43d4a9e04dc24fcf905f"
server_key_version: "encryptionkey@0"
I20260923 17:22:05.955895 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:05.967494 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:05.970537 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:05.970742 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:05.971323 6483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:05.972163 6483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:05.972234 6483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.972301 6483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:05.972347 6483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:05.981593 6483 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:41397
I20260923 17:22:05.981616 6617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:41397 every 8 connection(s)
I20260923 17:22:05.982744 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:05.985839 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 6483
I20260923 17:22:05.986325 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:34545
--local_ip_for_outbound_sockets=127.23.214.131
--tserver_master_addrs=127.23.214.190:42571
--webserver_port=44669
--webserver_interface=127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:22:05 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184125, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:05.998363 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.984638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:41771 (local address 127.23.214.190:42571)
0923 17:22:05.984773 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:05.984777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:05.985719 (+ 942us) server_negotiation.cc:407] Connection header received
0923 17:22:05.987327 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:05.987330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:05.987394 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:05.987490 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:05.988665 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.989539 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.990878 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.991365 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.991495 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.994188 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:05.994195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:05.994200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:05.994234 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:05.996049 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.996596 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.996602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.996605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.996706 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:05.997086 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:05.997093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:05.997097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:05.997411 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:05.997520 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:05.998049 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 17:22:05.998187 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:22:05.999240 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:05.985082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:41771)
0923 17:22:05.985524 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:05.985621 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:05.987111 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:05.987683 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:05.987690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:05.988038 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:05.988502 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.988516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.989680 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.989683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.990747 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:05.990758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:05.991547 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:05.991550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:05.991708 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:05.992381 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:05.992399 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:05.994006 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:05.996192 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.996197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.996209 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.996480 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.996820 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:05.996823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:05.996825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:05.996967 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:05.997518 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:05.997523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:05.997758 (+ 235us) client_negotiation.cc:769] Sending connection context
0923 17:22:05.997979 (+ 221us) client_negotiation.cc:240] Negotiation successful
0923 17:22:05.998251 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
I20260923 17:22:06.000427 6618 heartbeater.cc:347] Connected to a master server at 127.23.214.190:42571
I20260923 17:22:06.000684 6618 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:06.001250 6618 heartbeater.cc:511] Master 127.23.214.190:42571 requested a full tablet report, sending...
I20260923 17:22:06.002357 6285 ts_manager.cc:194] Registered new tserver with Master: 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397)
I20260923 17:22:06.002931 6285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:41771
W20260923 17:22:06.101015 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:06.101326 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:06.101402 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:06.105625 6621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:06.105705 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:06.105796 6621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:22:06.111140 6621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:34545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=44669
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:06.112257 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:06.113288 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:06.122771 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:06.122771 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:06.122779 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:06.123121 6621 server_base.cc:1034] running on GCE node
I20260923 17:22:06.123621 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:06.124240 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:06.125413 6621 hybrid_clock.cc:648] HybridClock initialized: now 1790184126125385 us; error 42 us; skew 500 ppm
Sep 23 17:22:06 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:06.128185 6621 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:22:06.129063 6621 webserver.cc:460] Webserver started at http://127.23.214.131:44669/ using document root <none> and password file <none>
I20260923 17:22:06.129624 6621 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:06.129727 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:06.133647 6621 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:06.135914 6636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:06.137260 6621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:22:06.137405 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "de85856704c0469e95f6a86bbd7d6719"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-9q7f"
server_key: "2f29f0f9981cdef3506b14eb0f8ebd77"
server_key_iv: "7d1e22e60c623f123d80649996dc11d4"
server_key_version: "encryptionkey@0"
I20260923 17:22:06.138078 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:06.179368 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:06.182294 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:06.182493 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:06.183090 6621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:06.183902 6621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:06.183949 6621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:06.184015 6621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:06.184055 6621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:06.192893 6621 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:34545
I20260923 17:22:06.192916 6749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:34545 every 8 connection(s)
I20260923 17:22:06.193910 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
Sep 23 17:22:06 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:06.203841 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 6621
I20260923 17:22:06.208551 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.195773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:56377 (local address 127.23.214.190:42571)
0923 17:22:06.195911 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.195915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.196771 (+ 856us) server_negotiation.cc:407] Connection header received
0923 17:22:06.198250 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.198253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.198312 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.198409 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:06.199562 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.200105 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.201230 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.201461 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.201571 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.204439 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:06.204444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:06.204447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:06.204478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:06.206395 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:06.207033 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:06.207036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:06.207037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:06.207115 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:06.207476 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:06.207479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:06.207480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:06.207721 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:06.207823 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.208292 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.208419 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:22:06.209414 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.196167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:56377)
0923 17:22:06.196631 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.196674 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:06.198039 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:06.198593 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:06.198600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:06.198957 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:06.199422 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.199435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.200227 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.200235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.201105 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.201114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.201614 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.201618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.201781 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.202404 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:06.202420 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:06.204222 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:06.206525 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:06.206530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:06.206541 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:06.206935 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:06.207202 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:06.207205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:06.207207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:06.207373 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:06.207825 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:06.207830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:06.208056 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 17:22:06.208216 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 17:22:06.208545 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260923 17:22:06.210883 6750 heartbeater.cc:347] Connected to a master server at 127.23.214.190:42571
I20260923 17:22:06.211148 6750 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:06.211639 6750 heartbeater.cc:511] Master 127.23.214.190:42571 requested a full tablet report, sending...
I20260923 17:22:06.212873 6285 ts_manager.cc:194] Registered new tserver with Master: de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545)
I20260923 17:22:06.213625 6285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:56377
I20260923 17:22:06.214048 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.204606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.23.214.190:42571)
0923 17:22:06.205006 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.205011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.205397 (+ 386us) server_negotiation.cc:407] Connection header received
0923 17:22:06.205444 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.205448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.205529 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.205629 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:06.206621 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.207380 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.208365 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.208679 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.208776 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.210371 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:06.210375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:06.210378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:06.210405 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:06.212162 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:06.212729 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:06.212733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:06.212735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:06.212821 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:06.213109 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:06.213113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:06.213115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:06.213302 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:06.213496 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.213728 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.213846 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":88,"threads_started":1}
I20260923 17:22:06.216068 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:22:06.220109 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.216684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.23.214.190:42571)
0923 17:22:06.216810 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.216815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.216963 (+ 148us) server_negotiation.cc:407] Connection header received
0923 17:22:06.217082 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.217084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.217166 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.217240 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:06.217406 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.217967 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.218857 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.219074 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.219150 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.219403 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:06.219747 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:06.219833 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.219920 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.219964 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:22:06.220705 6285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:06.223982 6285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:06.231907 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 1df39f32c30d41b6ae69045a62fad541 (DEFAULT_TABLE table=test-table [id=3787e3f86cc2427e9cf336ca3a97d66a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:06.232621 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df39f32c30d41b6ae69045a62fad541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:06.235654 6367 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Bootstrap starting.
I20260923 17:22:06.236618 6367 tablet_bootstrap.cc:654] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:06.236809 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.230791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34545 (local address 127.0.0.1:44172)
0923 17:22:06.230906 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.230922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:06.231000 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:06.231865 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:06.231868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:06.231887 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:06.232006 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.232012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.233817 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.233820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.235084 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.235098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.236368 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.236372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.236482 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.236526 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:22:06.236568 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 17:22:06.236634 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":4}
I20260923 17:22:06.237174 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.231669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.23.214.130:41397)
0923 17:22:06.232103 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.232117 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.232145 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:22:06.232226 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.232232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.232405 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.232579 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:06.232890 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.233973 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.235139 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.236120 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.236243 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.236372 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.236808 (+ 436us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.237003 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":87,"threads_started":1}
I20260923 17:22:06.237207 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.231235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:41397 (local address 127.0.0.1:50830)
0923 17:22:06.231764 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.231780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:06.231872 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:06.232553 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:06.232556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:06.232573 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:06.232769 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.232778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.234114 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.234120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.235019 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.235028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.236438 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.236442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.236574 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.236608 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:22:06.236696 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:22:06.236851 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":72,"thread_start_us":105,"threads_started":1}
I20260923 17:22:06.237664 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.230994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.23.214.131:34545)
0923 17:22:06.231330 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.231343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.231367 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:22:06.231458 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.231463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.231692 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.232185 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:06.232373 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.233573 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.235237 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.236240 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.237153 (+ 913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.237299 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.237383 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.237487 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":164,"threads_started":1}
I20260923 17:22:06.238715 6367 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: No bootstrap required, opened a new log
I20260923 17:22:06.238818 6367 ts_tablet_manager.cc:1397] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260923 17:22:06.239272 6367 raft_consensus.cc:348] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.239387 6367 raft_consensus.cc:374] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:06.239421 6367 raft_consensus.cc:729] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b172d24899374956bf3a9c3dea570f8f, State: Initialized, Role: FOLLOWER
I20260923 17:22:06.239550 6367 consensus_queue.cc:260] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.239827 6367 ts_tablet_manager.cc:1428] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 17:22:06.240659 6550 tablet_service.cc:1467] Processing CreateTablet for tablet 1df39f32c30d41b6ae69045a62fad541 (DEFAULT_TABLE table=test-table [id=3787e3f86cc2427e9cf336ca3a97d66a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:06.240658 6684 tablet_service.cc:1467] Processing CreateTablet for tablet 1df39f32c30d41b6ae69045a62fad541 (DEFAULT_TABLE table=test-table [id=3787e3f86cc2427e9cf336ca3a97d66a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:06.241801 6550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df39f32c30d41b6ae69045a62fad541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:06.241808 6684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df39f32c30d41b6ae69045a62fad541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:06.248895 6764 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Bootstrap starting.
I20260923 17:22:06.249621 6763 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Bootstrap starting.
I20260923 17:22:06.252218 6763 tablet_bootstrap.cc:654] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:06.252806 6764 tablet_bootstrap.cc:654] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:06.253677 6764 log.cc:824] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:06.253677 6763 log.cc:824] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:06.256238 6763 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: No bootstrap required, opened a new log
I20260923 17:22:06.256238 6764 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: No bootstrap required, opened a new log
I20260923 17:22:06.256480 6764 ts_tablet_manager.cc:1397] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 17:22:06.256480 6763 ts_tablet_manager.cc:1397] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:22:06.260737 6764 raft_consensus.cc:348] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.260737 6763 raft_consensus.cc:348] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.261081 6764 raft_consensus.cc:374] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:06.261081 6763 raft_consensus.cc:374] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:06.261147 6763 raft_consensus.cc:729] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de85856704c0469e95f6a86bbd7d6719, State: Initialized, Role: FOLLOWER
I20260923 17:22:06.261147 6764 raft_consensus.cc:729] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00da9567cbe6462594e5d66d9f5ab2dc, State: Initialized, Role: FOLLOWER
I20260923 17:22:06.261660 6764 consensus_queue.cc:260] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.261660 6763 consensus_queue.cc:260] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.262599 6618 heartbeater.cc:503] Master 127.23.214.190:42571 was elected leader, sending a full tablet report...
I20260923 17:22:06.262828 6750 heartbeater.cc:503] Master 127.23.214.190:42571 was elected leader, sending a full tablet report...
I20260923 17:22:06.262851 6764 ts_tablet_manager.cc:1428] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260923 17:22:06.262851 6763 ts_tablet_manager.cc:1428] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260923 17:22:06.291728 6479 tablet.cc:2369] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:06.386825 6768 raft_consensus.cc:482] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:06.387117 6768 raft_consensus.cc:504] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.388582 6768 leader_election.cc:288] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545), b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045)
I20260923 17:22:06.393780 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.388841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45045 (local address 127.23.214.130:51607)
0923 17:22:06.389104 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.389120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:06.389211 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:06.389595 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:06.389613 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:06.389632 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:06.389805 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.389810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.390871 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.390874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.392439 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.392445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.393368 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.393371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.393471 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.393507 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:22:06.393563 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 17:22:06.393614 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":221,"spinlock_wait_cycles":12032}
I20260923 17:22:06.393805 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.389088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:45941 (local address 127.23.214.131:34545)
0923 17:22:06.389221 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.389227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.389415 (+ 188us) server_negotiation.cc:407] Connection header received
0923 17:22:06.389471 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.389474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.389536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.389611 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:06.389953 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.390766 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.392580 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.393176 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.393315 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.393373 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.393580 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.393682 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:22:06.393815 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.389046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:51607 (local address 127.23.214.129:45045)
0923 17:22:06.389301 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.389306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.389318 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:22:06.389364 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.389366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.389458 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.389581 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:06.389931 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.390747 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.392580 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.393271 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.393375 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.393553 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.393591 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.393697 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260923 17:22:06.393788 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.388896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34545 (local address 127.23.214.130:45941)
0923 17:22:06.389319 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.389336 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:06.389406 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:06.389639 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:06.389641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:06.389676 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:06.389803 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.389810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.390871 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.390874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.392433 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:06.392445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.393294 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.393296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:06.393411 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.393439 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:22:06.393519 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:22:06.393607 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":146,"spinlock_wait_cycles":2816,"thread_start_us":135,"threads_started":1}
I20260923 17:22:06.394443 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df39f32c30d41b6ae69045a62fad541" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de85856704c0469e95f6a86bbd7d6719" is_pre_election: true
I20260923 17:22:06.394425 6414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df39f32c30d41b6ae69045a62fad541" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b172d24899374956bf3a9c3dea570f8f" is_pre_election: true
I20260923 17:22:06.394734 6414 raft_consensus.cc:2454] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 0.
I20260923 17:22:06.394734 6704 raft_consensus.cc:2454] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 0.
I20260923 17:22:06.395246 6504 leader_election.cc:302] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00da9567cbe6462594e5d66d9f5ab2dc, b172d24899374956bf3a9c3dea570f8f, de85856704c0469e95f6a86bbd7d6719; no voters:
I20260923 17:22:06.395567 6768 raft_consensus.cc:2781] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:06.395668 6768 raft_consensus.cc:482] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:06.395730 6768 raft_consensus.cc:3037] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:06.396757 6768 raft_consensus.cc:504] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.397104 6768 leader_election.cc:288] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 election: Requested vote from peers de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545), b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045)
I20260923 17:22:06.397467 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df39f32c30d41b6ae69045a62fad541" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de85856704c0469e95f6a86bbd7d6719"
I20260923 17:22:06.397599 6414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df39f32c30d41b6ae69045a62fad541" candidate_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b172d24899374956bf3a9c3dea570f8f"
I20260923 17:22:06.397653 6704 raft_consensus.cc:3037] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:06.397733 6414 raft_consensus.cc:3037] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:06.398991 6704 raft_consensus.cc:2454] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 1.
I20260923 17:22:06.398991 6414 raft_consensus.cc:2454] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00da9567cbe6462594e5d66d9f5ab2dc in term 1.
I20260923 17:22:06.399300 6503 leader_election.cc:302] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00da9567cbe6462594e5d66d9f5ab2dc, b172d24899374956bf3a9c3dea570f8f; no voters:
I20260923 17:22:06.399467 6768 raft_consensus.cc:2781] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:06.399686 6768 raft_consensus.cc:686] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 LEADER]: Becoming Leader. State: Replica: 00da9567cbe6462594e5d66d9f5ab2dc, State: Running, Role: LEADER
I20260923 17:22:06.400048 6768 consensus_queue.cc:237] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.403196 6284 catalog_manager.cc:5495] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130). New cstate: current_term: 1 leader_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260923 17:22:06.433320 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.429757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.23.214.130:41397)
0923 17:22:06.429905 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:06.429909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:06.429923 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:22:06.430120 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:06.430123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:06.430192 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:06.430285 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:06.430587 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.431074 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.431910 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:06.432119 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:06.432194 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:06.432495 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:06.432929 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:06.433017 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:06.433152 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 17:22:06.433205 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:22:06.439940 6704 raft_consensus.cc:1261] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Refusing update from remote peer 00da9567cbe6462594e5d66d9f5ab2dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:06.439935 6414 raft_consensus.cc:1261] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Refusing update from remote peer 00da9567cbe6462594e5d66d9f5ab2dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:06.440685 6768 consensus_queue.cc:1035] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:06.440886 6771 consensus_queue.cc:1035] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:22:06.444952 6751 tablet.cc:2369] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:06.451145 6774 mvcc.cc:205] Tried to move back new op lower bound from 7332594181890183168 to 7332594181737078784. Current Snapshot: MvccSnapshot[applied={T|T < 7332594181890183168 or (T in {7332594181890183168})}]
I20260923 17:22:06.458369 6285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:06.458567 6285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:06.461351 6285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Sending DeleteTablet for 3 replicas of tablet 1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:06.462260 6684 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df39f32c30d41b6ae69045a62fad541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:06 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:44172
I20260923 17:22:06.462265 6550 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df39f32c30d41b6ae69045a62fad541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:06 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:50830
I20260923 17:22:06.462337 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df39f32c30d41b6ae69045a62fad541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:06 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:38702
I20260923 17:22:06.462841 6783 tablet_replica.cc:330] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: stopping tablet replica
I20260923 17:22:06.462996 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 6324
I20260923 17:22:06.462999 6784 tablet_replica.cc:330] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: stopping tablet replica
I20260923 17:22:06.462987 6783 raft_consensus.cc:2229] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:06.463380 6784 raft_consensus.cc:2229] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:06.463922 6784 raft_consensus.cc:2258] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:06.465390 6784 ts_tablet_manager.cc:1905] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:06.467886 6784 ts_tablet_manager.cc:1918] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:06.468024 6784 log.cc:1196] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:06.468328 6784 ts_tablet_manager.cc:1939] T 1df39f32c30d41b6ae69045a62fad541 P 00da9567cbe6462594e5d66d9f5ab2dc: Deleting consensus metadata
I20260923 17:22:06.469516 6271 catalog_manager.cc:4841] TS 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397): tablet 1df39f32c30d41b6ae69045a62fad541 (table test-table [id=3787e3f86cc2427e9cf336ca3a97d66a]) successfully deleted
I20260923 17:22:06.470116 6785 tablet_replica.cc:330] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: stopping tablet replica
I20260923 17:22:06.470466 6785 raft_consensus.cc:2229] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:06.470808 6785 raft_consensus.cc:2258] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 17:22:06.471774 6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:45045 (error 108)
I20260923 17:22:06.472005 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 6483
I20260923 17:22:06.473551 6785 ts_tablet_manager.cc:1905] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:06.475787 6785 ts_tablet_manager.cc:1918] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:06.475873 6785 log.cc:1196] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:06.476205 6785 ts_tablet_manager.cc:1939] T 1df39f32c30d41b6ae69045a62fad541 P de85856704c0469e95f6a86bbd7d6719: Deleting consensus metadata
I20260923 17:22:06.477370 6270 catalog_manager.cc:4841] TS de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545): tablet 1df39f32c30d41b6ae69045a62fad541 (table test-table [id=3787e3f86cc2427e9cf336ca3a97d66a]) successfully deleted
I20260923 17:22:06.478722 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 6621
I20260923 17:22:06.479048 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.478740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45045 (local address 127.0.0.1:38714)
0923 17:22:06.478873 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.478951 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:22:06.479471 6269 catalog_manager.cc:4584] TS b172d24899374956bf3a9c3dea570f8f (127.23.214.129:45045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1df39f32c30d41b6ae69045a62fad541: Network error: Client connection negotiation failed: client connection to 127.23.214.129:45045: connect: Connection refused (error 111)
I20260923 17:22:06.484647 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 6252
I20260923 17:22:06.491122 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42571
--webserver_interface=127.23.214.190
--webserver_port=34751
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:06.602788 6787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:06.603087 6787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:06.603174 6787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:06.607146 6787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:06.607230 6787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:06.607270 6787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:06.607306 6787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:06.607342 6787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:06.612541 6787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=34751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:06.613579 6787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:06.614637 6787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:06.623025 6792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:06.623179 6795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:06.623039 6793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:06.623091 6787 server_base.cc:1034] running on GCE node
I20260923 17:22:06.623797 6787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:06.624917 6787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:06.626128 6787 hybrid_clock.cc:648] HybridClock initialized: now 1790184126626104 us; error 36 us; skew 500 ppm
Sep 23 17:22:06 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:06.628808 6787 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:22:06.629622 6787 webserver.cc:460] Webserver started at http://127.23.214.190:34751/ using document root <none> and password file <none>
I20260923 17:22:06.630215 6787 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:06.630278 6787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:06.634140 6787 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 17:22:06.636464 6802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:06.637830 6787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:06.637996 6787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "9cc7d3320ff3462685c6ffdab493e27f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "e58b6e0f0a9563f3c6a954907defb924"
server_key_iv: "ceb7f72eb6b1d49bec9690d4a57e6bae"
server_key_version: "encryptionkey@0"
I20260923 17:22:06.638556 6787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:06.649219 6787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:06.652410 6787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:06.652580 6787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:06.659442 6787 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:42571
I20260923 17:22:06.659735 6854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:42571 every 8 connection(s)
I20260923 17:22:06.660534 6787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:06.664737 6855 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:22:06.666421 6855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Bootstrap starting.
I20260923 17:22:06.667140 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 6787
I20260923 17:22:06.667647 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:45045
--local_ip_for_outbound_sockets=127.23.214.129
--tserver_master_addrs=127.23.214.190:42571
--webserver_port=38611
--webserver_interface=127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:06.688674 6855 log.cc:824] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:06.702169 6855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:06.702667 6855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f: Bootstrap complete.
I20260923 17:22:06.707293 6855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:06.707707 6855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc7d3320ff3462685c6ffdab493e27f, State: Initialized, Role: FOLLOWER
I20260923 17:22:06.708241 6855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:06.708362 6855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:06.708454 6855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:06.708581 6855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:22:06.710698 6855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:06.711053 6855 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc7d3320ff3462685c6ffdab493e27f; no voters:
I20260923 17:22:06.711400 6855 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:22:06.711496 6859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:22:06.711705 6859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [term 3 LEADER]: Becoming Leader. State: Replica: 9cc7d3320ff3462685c6ffdab493e27f, State: Running, Role: LEADER
I20260923 17:22:06.712033 6859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } }
I20260923 17:22:06.712769 6855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:06.713238 6859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc7d3320ff3462685c6ffdab493e27f. Latest consensus state: current_term: 3 leader_uuid: "9cc7d3320ff3462685c6ffdab493e27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } } }
I20260923 17:22:06.713361 6859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:06.713768 6863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:06.713891 6860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9cc7d3320ff3462685c6ffdab493e27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc7d3320ff3462685c6ffdab493e27f" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 42571 } } }
I20260923 17:22:06.714103 6860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc7d3320ff3462685c6ffdab493e27f [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:06.719906 6863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:06.720666 6863 catalog_manager.cc:1259] Loaded cluster ID: 8a976f0a29e34d57bce1c0b16f2d080c
I20260923 17:22:06.720736 6863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 17:22:06.721932 6863 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f72a4b839ad google::LogMessage::Fail() at ??:0
@ 0x7f72a4b8688a google::LogMessage::SendToLog() at ??:0
@ 0x7f72a4b834de google::LogMessage::Flush() at ??:0
@ 0x7f72a4b84a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f72af4c2fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f72af4c3d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f72af4c0579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f72af504640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f72a6dd53d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f72a6dd5c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f72a6dd73f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f72a6dc6512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f72a840a6db start_thread at ??:0
@ 0x7f72a368161f clone at ??:0
W20260923 17:22:06.822667 6857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:06.823128 6857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:06.823230 6857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:06.829653 6857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:06.829759 6857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:06.829909 6857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:22:06.838286 6857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:45045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=38611
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:06.839715 6857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:06.841099 6857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:06.853180 6896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:06.853251 6897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:06.853595 6899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:06.853636 6857 server_base.cc:1034] running on GCE node
I20260923 17:22:06.854153 6857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:06.855019 6857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:06.856225 6857 hybrid_clock.cc:648] HybridClock initialized: now 1790184126856199 us; error 34 us; skew 500 ppm
Sep 23 17:22:06 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:06.860034 6857 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:22:06.861528 6857 webserver.cc:460] Webserver started at http://127.23.214.129:38611/ using document root <none> and password file <none>
I20260923 17:22:06.862322 6857 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:06.862421 6857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:06.867879 6857 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260923 17:22:06.870214 6906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:06.871588 6857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 17:22:06.871744 6857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b172d24899374956bf3a9c3dea570f8f"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "bfb434c2e33c385455e06fdb34536911"
server_key_iv: "de621cce6a5ac91b77a27673e3de4738"
server_key_version: "encryptionkey@0"
I20260923 17:22:06.872311 6857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:06.903419 6857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:06.906924 6857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:06.907142 6857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:06.907760 6857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:06.908905 6913 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:22:06.914606 6857 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:22:06.914693 6857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 17:22:06.914783 6857 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:22:06.916566 6857 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:22:06.916639 6857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 17:22:06.916942 6913 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Bootstrap starting.
I20260923 17:22:06.927115 6857 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:45045
I20260923 17:22:06.928390 6857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:06.928555 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:45045 every 8 connection(s)
I20260923 17:22:06.936029 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 6857
I20260923 17:22:06.936528 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:41397
--local_ip_for_outbound_sockets=127.23.214.130
--tserver_master_addrs=127.23.214.190:42571
--webserver_port=33945
--webserver_interface=127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:06.939041 6913 log.cc:824] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:06.947630 6913 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:06.948016 6913 tablet_bootstrap.cc:492] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Bootstrap complete.
I20260923 17:22:06.949132 6913 ts_tablet_manager.cc:1397] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.008s
I20260923 17:22:06.951663 6913 raft_consensus.cc:348] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.952147 6913 raft_consensus.cc:729] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b172d24899374956bf3a9c3dea570f8f, State: Initialized, Role: FOLLOWER
I20260923 17:22:06.952652 6913 consensus_queue.cc:260] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:06.953624 6913 ts_tablet_manager.cc:1428] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260923 17:22:06.990942 7025 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:22:06.931339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:48675)
0923 17:22:06.931950 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.932010 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:06.933286 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:06.990210 (+ 56924us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":167,"threads_started":1}
W20260923 17:22:06.991528 7023 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:42571 (0 consecutive failures): Network error: Failed to ping master at 127.23.214.190:42571: Client connection negotiation failed: client connection to 127.23.214.190:42571: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:22:06.992334 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.991992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:39433)
0923 17:22:06.992157 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.992247 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:22:06.993027 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:06.992786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:39581)
0923 17:22:06.992905 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:06.992963 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:22:06.993217 7023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:22:07.076648 7026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:07.077003 7026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:07.077092 7026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:07.081276 7026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:07.081370 7026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:07.081458 7026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:22:07.086802 7026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:41397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=33945
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:07.087916 7026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:07.088972 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:07.098716 7034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:07.098711 7033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:07.098704 7036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:07.099113 7026 server_base.cc:1034] running on GCE node
I20260923 17:22:07.099547 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:07.100266 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:07.101437 7026 hybrid_clock.cc:648] HybridClock initialized: now 1790184127101410 us; error 38 us; skew 500 ppm
Sep 23 17:22:07 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:07.104421 7026 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:22:07.105378 7026 webserver.cc:460] Webserver started at http://127.23.214.130:33945/ using document root <none> and password file <none>
I20260923 17:22:07.106041 7026 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:07.106138 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:07.110087 7026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:22:07.112219 7043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:07.113443 7026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:07.113600 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "00da9567cbe6462594e5d66d9f5ab2dc"
format_stamp: "Formatted at 2026-09-23 17:22:04 on dist-test-slave-9q7f"
server_key: "dd709e3494e4ad76c566111578472cd1"
server_key_iv: "dad7f739f75e43d4a9e04dc24fcf905f"
server_key_version: "encryptionkey@0"
I20260923 17:22:07.114200 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:07.139416 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:07.143143 7026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:07.143378 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:07.144095 7026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:07.144991 7026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:07.145058 7026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:07.145125 7026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:07.145171 7026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:07.155082 7026 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:41397
I20260923 17:22:07.155118 7157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:41397 every 8 connection(s)
I20260923 17:22:07.156124 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:07.160005 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.158285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:42579)
0923 17:22:07.158718 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:07.158835 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
W20260923 17:22:07.160498 7158 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:42571 (0 consecutive failures): Network error: Failed to ping master at 127.23.214.190:42571: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
I20260923 17:22:07.161091 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.160842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:50089)
0923 17:22:07.160979 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:07.161034 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:22:07.161523 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7026
I20260923 17:22:07.161834 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.161611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:41969)
0923 17:22:07.161726 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:07.161782 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:22:07.162027 7158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:22:07.162019 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:34545
--local_ip_for_outbound_sockets=127.23.214.131
--tserver_master_addrs=127.23.214.190:42571
--webserver_port=44669
--webserver_interface=127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:07.280030 7161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:07.280319 7161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:07.280403 7161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:07.284507 7161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:07.284579 7161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:07.284684 7161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:22:07.290066 7161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:34545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=44669
--tserver_master_addrs=127.23.214.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:07.291219 7161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:07.292272 7161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:22:07.302151 7161 server_base.cc:1034] running on GCE node
W20260923 17:22:07.302079 7167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:07.302088 7170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:07.302088 7168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:07.302765 7161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:07.303386 7161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:07.304543 7161 hybrid_clock.cc:648] HybridClock initialized: now 1790184127304537 us; error 47 us; skew 500 ppm
Sep 23 17:22:07 dist-test-slave-9q7f krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184127, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:07.307394 7161 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:22:07.308346 7161 webserver.cc:460] Webserver started at http://127.23.214.131:44669/ using document root <none> and password file <none>
I20260923 17:22:07.308897 7161 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:07.308983 7161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:07.312805 7161 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:07.314864 7178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:07.316049 7161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:07.316179 7161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "de85856704c0469e95f6a86bbd7d6719"
format_stamp: "Formatted at 2026-09-23 17:22:05 on dist-test-slave-9q7f"
server_key: "2f29f0f9981cdef3506b14eb0f8ebd77"
server_key_iv: "7d1e22e60c623f123d80649996dc11d4"
server_key_version: "encryptionkey@0"
I20260923 17:22:07.316711 7161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:07.326876 7161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:07.330226 7161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:07.330426 7161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:07.331032 7161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:07.331866 7161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:07.331935 7161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:07.332008 7161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:07.332047 7161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:07.342057 7161 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:34545
I20260923 17:22:07.342079 7292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:34545 every 8 connection(s)
I20260923 17:22:07.343123 7161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:07.347556 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.345655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:54779)
0923 17:22:07.346222 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:07.346341 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":105,"threads_started":1}
W20260923 17:22:07.348107 7293 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:42571 (0 consecutive failures): Network error: Failed to ping master at 127.23.214.190:42571: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
I20260923 17:22:07.348423 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7161
I20260923 17:22:07.348778 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.348539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:59521)
0923 17:22:07.348663 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:07.348715 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 17:22:07.349732 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.349529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:34115)
0923 17:22:07.349626 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:07.349672 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 17:22:07.349958 7293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:22:07.994324 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:07.993892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:49941)
0923 17:22:07.994173 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:07.994229 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
I20260923 17:22:08.122429 7298 raft_consensus.cc:482] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:08.122682 7298 raft_consensus.cc:504] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:08.123817 7298 leader_election.cc:288] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545), 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397)
Sep 23 17:22:08 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.131@KRBTEST.COM
Sep 23 17:22:08 dist-test-slave-9q7f krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.130@KRBTEST.COM
I20260923 17:22:08.138175 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.124063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:41397 (local address 127.23.214.129:47295)
0923 17:22:08.124408 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:08.124422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:08.124525 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:08.126018 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:08.126023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:08.126533 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:08.127034 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:08.127065 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.128346 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.128355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:08.129328 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:08.129338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.130115 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.130118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:08.130320 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:08.131090 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:08.131103 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:08.133290 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:08.135612 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:08.135614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:08.135616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:08.135903 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:08.136286 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:08.136289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:08.136292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:08.136442 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:08.137127 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:08.137131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:08.137368 (+ 237us) client_negotiation.cc:769] Sending connection context
0923 17:22:08.137557 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 17:22:08.137974 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":153,"spinlock_wait_cycles":13312,"thread_start_us":77,"threads_started":1}
I20260923 17:22:08.138190 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.124236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:47835 (local address 127.23.214.131:34545)
0923 17:22:08.124581 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:08.124593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:08.124629 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:22:08.125275 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:08.125280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:08.125626 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:08.125941 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:08.127213 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.128197 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.129441 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.129999 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.130172 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:08.133209 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:08.133217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:08.133238 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:08.133275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:08.135314 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:08.136055 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:08.136060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:08.136064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:08.136197 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:08.136550 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:08.136553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:08.136555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:08.136988 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:08.137127 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:08.137724 (+ 597us) server_negotiation.cc:299] Negotiation successful
0923 17:22:08.138005 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260923 17:22:08.138234 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.124231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:47295 (local address 127.23.214.130:41397)
0923 17:22:08.124640 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:08.124650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:08.124681 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:22:08.125276 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:08.125280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:08.125628 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:08.125937 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:08.127225 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.128196 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.129472 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.129992 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.130162 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:08.133479 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:08.133485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:08.133496 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:08.133524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:08.135485 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:08.136034 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:08.136046 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:08.136051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:08.136178 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:08.136548 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:08.136555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:08.136557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:08.136993 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:08.137143 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:08.137684 (+ 541us) server_negotiation.cc:299] Negotiation successful
0923 17:22:08.138044 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":78,"threads_started":1}
I20260923 17:22:08.138219 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.123993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34545 (local address 127.23.214.129:47835)
0923 17:22:08.124190 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:08.124207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:08.124314 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:08.126017 (+ 1703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:08.126023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:08.126533 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:08.127034 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:08.127049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.128347 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.128350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:08.129321 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:08.129331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:08.130138 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:08.130141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:08.130317 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:08.131063 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:08.131092 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:08.132915 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:08.135463 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:08.135470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:08.135485 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:08.135910 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:08.136318 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:08.136321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:08.136323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:08.136446 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:08.137106 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:08.137112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:08.137368 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 17:22:08.137593 (+ 225us) client_negotiation.cc:240] Negotiation successful
0923 17:22:08.138054 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":334,"spinlock_wait_cycles":22912}
W20260923 17:22:08.140810 6909 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
W20260923 17:22:08.140982 6908 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:08.141077 6908 leader_election.cc:302] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b172d24899374956bf3a9c3dea570f8f; no voters: 00da9567cbe6462594e5d66d9f5ab2dc, de85856704c0469e95f6a86bbd7d6719
I20260923 17:22:08.141366 7298 raft_consensus.cc:2726] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:22:08.163087 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.162629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:57285)
0923 17:22:08.162902 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:08.162965 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260923 17:22:08.350980 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.350534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:45155)
0923 17:22:08.350794 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:08.350867 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":109,"threads_started":1}
I20260923 17:22:08.995476 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:08.995078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:35033)
0923 17:22:08.995325 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:08.995379 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260923 17:22:09.164301 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:09.163839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:44857)
0923 17:22:09.164115 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:09.164176 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260923 17:22:09.352170 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:09.351699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:46443)
0923 17:22:09.351988 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:09.352057 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260923 17:22:09.827503 7307 raft_consensus.cc:482] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:09.827651 7307 raft_consensus.cc:504] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:09.828310 7307 leader_election.cc:288] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545), 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397)
W20260923 17:22:09.829133 6908 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
W20260923 17:22:09.829214 6909 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:09.829262 6908 leader_election.cc:302] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b172d24899374956bf3a9c3dea570f8f; no voters: 00da9567cbe6462594e5d66d9f5ab2dc, de85856704c0469e95f6a86bbd7d6719
I20260923 17:22:09.829433 7307 raft_consensus.cc:2726] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:22:09.996932 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:09.996426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:47469)
0923 17:22:09.996750 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:09.996818 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
I20260923 17:22:10.165679 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.165176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:33181)
0923 17:22:10.165494 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.165584 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":82,"threads_started":1}
I20260923 17:22:10.353593 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.353030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:58643)
0923 17:22:10.353291 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.353355 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260923 17:22:10.998375 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:10.997934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:56773)
0923 17:22:10.998203 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:10.998276 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260923 17:22:11.167241 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.166666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:33439)
0923 17:22:11.167006 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.167101 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
I20260923 17:22:11.355050 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.354582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:50365)
0923 17:22:11.354879 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.354947 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260923 17:22:12.000020 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:11.999458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:34115)
0923 17:22:11.999829 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:11.999916 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":160,"threads_started":1}
I20260923 17:22:12.088631 7315 raft_consensus.cc:482] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:12.088759 7315 raft_consensus.cc:504] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:12.089119 7315 leader_election.cc:288] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545), 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397)
W20260923 17:22:12.090070 6909 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
W20260923 17:22:12.090176 6908 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:12.090289 6908 leader_election.cc:302] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b172d24899374956bf3a9c3dea570f8f; no voters: 00da9567cbe6462594e5d66d9f5ab2dc, de85856704c0469e95f6a86bbd7d6719
I20260923 17:22:12.090446 7315 raft_consensus.cc:2726] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:22:12.168578 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.168092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:53665)
0923 17:22:12.168385 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:12.168457 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260923 17:22:12.356415 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:12.355977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:44691)
0923 17:22:12.356251 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:12.356314 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260923 17:22:13.001185 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.000743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:45283)
0923 17:22:13.001017 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.001086 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260923 17:22:13.169793 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.169367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:59431)
0923 17:22:13.169633 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.169693 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260923 17:22:13.357697 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:13.357244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:47327)
0923 17:22:13.357536 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:13.357599 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260923 17:22:14.002475 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:14.002059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:43847)
0923 17:22:14.002319 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:14.002380 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260923 17:22:14.171110 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:14.170612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:47339)
0923 17:22:14.170934 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:14.170998 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":73,"threads_started":1}
I20260923 17:22:14.359257 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:14.358709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:57863)
0923 17:22:14.359111 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:14.359164 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":140,"threads_started":1}
I20260923 17:22:15.003800 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.003319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:35563)
0923 17:22:15.003629 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.003691 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260923 17:22:15.172330 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.171896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:52257)
0923 17:22:15.172177 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.172236 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
I20260923 17:22:15.360481 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:15.360034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:58527)
0923 17:22:15.360319 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:15.360378 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260923 17:22:15.492952 7328 raft_consensus.cc:482] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:15.493135 7328 raft_consensus.cc:504] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:15.493620 7328 leader_election.cc:288] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545), 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397)
W20260923 17:22:15.494493 6908 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
W20260923 17:22:15.494609 6909 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:15.494726 6909 leader_election.cc:302] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b172d24899374956bf3a9c3dea570f8f; no voters: 00da9567cbe6462594e5d66d9f5ab2dc, de85856704c0469e95f6a86bbd7d6719
I20260923 17:22:15.494905 7328 raft_consensus.cc:2726] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:22:16.005070 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:16.004632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:44881)
0923 17:22:16.004896 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:16.004955 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260923 17:22:16.173622 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:16.173105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:37057)
0923 17:22:16.173414 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:16.173480 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260923 17:22:16.361858 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:16.361356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:39755)
0923 17:22:16.361699 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:16.361761 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260923 17:22:17.006708 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:17.006199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:45755)
0923 17:22:17.006503 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:17.006594 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260923 17:22:17.174952 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:17.174488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:53515)
0923 17:22:17.174795 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:17.174855 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
I20260923 17:22:17.290689 7334 raft_consensus.cc:482] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:17.290820 7334 raft_consensus.cc:504] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de85856704c0469e95f6a86bbd7d6719" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34545 } } peers { permanent_uuid: "00da9567cbe6462594e5d66d9f5ab2dc" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 41397 } } peers { permanent_uuid: "b172d24899374956bf3a9c3dea570f8f" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45045 } }
I20260923 17:22:17.291216 7334 leader_election.cc:288] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545), 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397)
W20260923 17:22:17.292038 6909 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da9567cbe6462594e5d66d9f5ab2dc (127.23.214.130:41397): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
W20260923 17:22:17.292124 6908 leader_election.cc:341] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de85856704c0469e95f6a86bbd7d6719 (127.23.214.131:34545): Not found: Tablet not found: 1df39f32c30d41b6ae69045a62fad541
I20260923 17:22:17.292164 6909 leader_election.cc:302] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b172d24899374956bf3a9c3dea570f8f; no voters: 00da9567cbe6462594e5d66d9f5ab2dc, de85856704c0469e95f6a86bbd7d6719
I20260923 17:22:17.292327 7334 raft_consensus.cc:2726] T 1df39f32c30d41b6ae69045a62fad541 P b172d24899374956bf3a9c3dea570f8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:22:17.363298 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:17.362842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:36909)
0923 17:22:17.363144 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:17.363202 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
I20260923 17:22:18.008041 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.007616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:44217)
0923 17:22:18.007877 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:18.007941 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260923 17:22:18.176069 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.175681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:36751)
0923 17:22:18.175918 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:18.175972 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260923 17:22:18.364626 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:18.364147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:51261)
0923 17:22:18.364418 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:18.364524 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20260923 17:22:19.009452 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.008946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:46547)
0923 17:22:19.009284 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:19.009351 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260923 17:22:19.177404 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.176892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:44183)
0923 17:22:19.177242 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:19.177304 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":78,"threads_started":1}
I20260923 17:22:19.366055 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:19.365482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:56819)
0923 17:22:19.365804 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:19.365925 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
I20260923 17:22:20.010735 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:20.010324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:58513)
0923 17:22:20.010577 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:20.010636 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260923 17:22:20.178642 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:20.178231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:46453)
0923 17:22:20.178466 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:20.178519 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260923 17:22:20.367348 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:20.366918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:38033)
0923 17:22:20.367190 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:20.367251 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260923 17:22:21.012362 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.011754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:34721)
0923 17:22:21.012142 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.012223 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":98,"threads_started":1}
I20260923 17:22:21.179842 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.179423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:55153)
0923 17:22:21.179679 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.179745 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260923 17:22:21.368584 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:21.368151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.131:36015)
0923 17:22:21.368409 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:21.368487 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260923 17:22:22.013970 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.013425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.129:39539)
0923 17:22:22.013727 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.013791 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
I20260923 17:22:22.181257 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:22.180807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:42571 (local address 127.23.214.130:55731)
0923 17:22:22.181052 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:22.181129 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260923 17:22:22.356956 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 6857
I20260923 17:22:22.363063 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7026
I20260923 17:22:22.368569 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7161
2026-09-23T17:22:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19305 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:22:22 dist-test-slave-9q7f krb5kdc[7353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:22:22 dist-test-slave-9q7f krb5kdc[7353](info): set up 2 sockets
Sep 23 17:22:22 dist-test-slave-9q7f krb5kdc[7353](info): commencing operation
krb5kdc: starting...
W20260923 17:22:23.420220 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:22:23 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:22:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:22:23.541854 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34557
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:34557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:23.657410 7366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:23.657784 7366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:23.657904 7366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:23.662052 7366 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:22:23.662161 7366 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:22:23.662194 7366 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:22:23.662230 7366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:23.662279 7366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:23.662329 7366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:23.662374 7366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:23.662418 7366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:23.668072 7366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:34557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:23.669330 7366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:23.670516 7366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:22:23.679773 7366 server_base.cc:1034] running on GCE node
W20260923 17:22:23.679728 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:23.679733 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:23.679750 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:23.680740 7366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:23.681788 7366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:23.682957 7366 hybrid_clock.cc:648] HybridClock initialized: now 1790184143682940 us; error 29 us; skew 500 ppm
Sep 23 17:22:23 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184143, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:23.685760 7366 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:22:23.686679 7366 webserver.cc:460] Webserver started at http://127.23.214.190:46221/ using document root <none> and password file <none>
I20260923 17:22:23.687220 7366 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:23.687278 7366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:23.687516 7366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:23.689517 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "48f387240dfc4745a8575390ac9394a6"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "bcdd600db2659ced1bfa2a4c2933c5d6"
server_key_iv: "34743d09aa8a3db516f509fd663f4746"
server_key_version: "encryptionkey@0"
I20260923 17:22:23.690117 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "48f387240dfc4745a8575390ac9394a6"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "bcdd600db2659ced1bfa2a4c2933c5d6"
server_key_iv: "34743d09aa8a3db516f509fd663f4746"
server_key_version: "encryptionkey@0"
I20260923 17:22:23.693987 7366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:22:23.696318 7381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:23.697573 7366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:23.697697 7366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "48f387240dfc4745a8575390ac9394a6"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "bcdd600db2659ced1bfa2a4c2933c5d6"
server_key_iv: "34743d09aa8a3db516f509fd663f4746"
server_key_version: "encryptionkey@0"
I20260923 17:22:23.697789 7366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:23.722635 7366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:23.725617 7366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:23.725782 7366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:23.733223 7366 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:34557
I20260923 17:22:23.733572 7434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:34557 every 8 connection(s)
I20260923 17:22:23.734262 7366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:23.737241 7435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:23.738384 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7366
I20260923 17:22:23.738529 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:22:23.738837 24410 external_mini_cluster.cc:1428] Setting key 96f74a27984fb6c731d000660319effc
I20260923 17:22:23.743441 7435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Bootstrap starting.
I20260923 17:22:23.745683 7435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:23.746498 7435 log.cc:824] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:22:23 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:23.748479 7435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: No bootstrap required, opened a new log
I20260923 17:22:23.752071 7435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:23.752332 7435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:23.752504 7435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f387240dfc4745a8575390ac9394a6, State: Initialized, Role: FOLLOWER
I20260923 17:22:23.753144 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:23.740280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.23.214.190:34557)
0923 17:22:23.740735 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:23.740750 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:23.740786 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:22:23.741427 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:23.741442 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:23.741794 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:23.742190 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:23.743002 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:23.744031 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:23.744874 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:23.745347 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:23.745526 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:23.748121 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:23.748126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:23.748139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:23.748164 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:23.749986 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:23.750487 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:23.750493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:23.750507 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:23.750621 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:23.750899 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:23.750903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:23.750905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:23.751291 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:23.751504 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:23.751768 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 17:22:23.752049 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":136,"threads_started":1}
I20260923 17:22:23.753130 7435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:23.753331 7435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:23.753403 7435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:23.753544 7435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:23.754736 7435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:23.755108 7435 leader_election.cc:302] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f387240dfc4745a8575390ac9394a6; no voters:
I20260923 17:22:23.755466 7435 leader_election.cc:288] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:22:23.755527 7440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:23.756016 7440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 LEADER]: Becoming Leader. State: Replica: 48f387240dfc4745a8575390ac9394a6, State: Running, Role: LEADER
I20260923 17:22:23.756318 7440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:23.756752 7435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:23.757979 7441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f387240dfc4745a8575390ac9394a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } } }
I20260923 17:22:23.758152 7441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:23.758507 7442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f387240dfc4745a8575390ac9394a6. Latest consensus state: current_term: 1 leader_uuid: "48f387240dfc4745a8575390ac9394a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } } }
I20260923 17:22:23.758597 7442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:23.758687 7449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:23.762461 7449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:23.767596 7449 catalog_manager.cc:1347] Generated new cluster ID: 461b41e05c5548dcaa009e67d615fa70
I20260923 17:22:23.767661 7449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:23.782174 7449 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:22:23.783135 7449 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:23.788470 7449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Generated new TSK 0
I20260923 17:22:23.789217 7449 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:22:23.829421 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34557
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:23.943145 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:23.943476 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:23.943580 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:23.947751 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:23.947862 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:23.947973 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:22:23.953466 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:23.954706 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:23.955835 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:23.966854 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:23.966840 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:23.966840 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:23.967285 7462 server_base.cc:1034] running on GCE node
I20260923 17:22:23.967783 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:23.968474 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:23.969655 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790184143969635 us; error 28 us; skew 500 ppm
Sep 23 17:22:23 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184143, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:23.972738 7462 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:22:23.973628 7462 webserver.cc:460] Webserver started at http://127.23.214.129:36005/ using document root <none> and password file <none>
I20260923 17:22:23.974253 7462 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:23.974318 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:23.974484 7462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:23.976660 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "b37c2506020546d7894d35db6be03f7e"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "d3382f16e52273f4d52996414c836944"
server_key_iv: "45a2552f76a53cd61c5db172032a65da"
server_key_version: "encryptionkey@0"
I20260923 17:22:23.977301 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "b37c2506020546d7894d35db6be03f7e"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "d3382f16e52273f4d52996414c836944"
server_key_iv: "45a2552f76a53cd61c5db172032a65da"
server_key_version: "encryptionkey@0"
I20260923 17:22:23.981524 7462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260923 17:22:23.984194 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:23.985524 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:23.985698 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b37c2506020546d7894d35db6be03f7e"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "d3382f16e52273f4d52996414c836944"
server_key_iv: "45a2552f76a53cd61c5db172032a65da"
server_key_version: "encryptionkey@0"
I20260923 17:22:23.985850 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:24.003345 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:24.006842 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:24.007042 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:24.007653 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:24.008438 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:24.008481 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.008546 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:24.008584 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.017766 7462 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:38519
I20260923 17:22:24.017787 7591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:38519 every 8 connection(s)
I20260923 17:22:24.018676 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:24.026638 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7462
I20260923 17:22:24.026777 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:22:24.027092 24410 external_mini_cluster.cc:1428] Setting key f912053ccf0859deff03bc6b66a9436e
Sep 23 17:22:24 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184143, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:24.032725 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.020413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:44033 (local address 127.23.214.190:34557)
0923 17:22:24.020573 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.020577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.021422 (+ 845us) server_negotiation.cc:407] Connection header received
0923 17:22:24.022787 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.022790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.022836 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.022910 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:24.024042 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.024589 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.025668 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.025944 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.026031 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.028834 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:24.028837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:24.028839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:24.028861 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:24.030425 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.031028 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.031031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.031032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.031119 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.031503 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.031505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.031506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.031752 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:24.031853 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.032333 (+ 480us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.032486 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:22:24.033411 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.020839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:44033)
0923 17:22:24.021274 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.021316 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.022568 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.023074 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.023081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.023425 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:24.023916 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.023928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.024696 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.024699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.025553 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.025562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.026078 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.026081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.026215 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.026898 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:24.026917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:24.028668 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:24.030538 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:24.030546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:24.030563 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:24.030930 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:24.031208 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:24.031211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:24.031214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:24.031408 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:24.031857 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:24.031862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:24.032094 (+ 232us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.032250 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.032495 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260923 17:22:24.034762 7592 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34557
I20260923 17:22:24.035038 7592 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:24.035640 7592 heartbeater.cc:511] Master 127.23.214.190:34557 requested a full tablet report, sending...
I20260923 17:22:24.037443 7398 ts_manager.cc:194] Registered new tserver with Master: b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519)
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:22:24.038743 7398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:44033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:22:24.056676 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34557
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:24.168041 7597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:24.168352 7597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:24.168426 7597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:24.172633 7597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:24.172698 7597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:24.172793 7597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:22:24.178362 7597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:24.179530 7597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:24.180612 7597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:24.190497 7603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:24.190586 7602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:24.190497 7605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:24.190853 7597 server_base.cc:1034] running on GCE node
I20260923 17:22:24.191267 7597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:24.191998 7597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:24.193192 7597 hybrid_clock.cc:648] HybridClock initialized: now 1790184144193176 us; error 34 us; skew 500 ppm
Sep 23 17:22:24 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184144, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:24.196211 7597 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:22:24.197023 7597 webserver.cc:460] Webserver started at http://127.23.214.130:45863/ using document root <none> and password file <none>
I20260923 17:22:24.197552 7597 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:24.197615 7597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:24.197781 7597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:24.199831 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "b8dddce61c854580bd646a4b4c4287c5"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "72a20d48a5013ec3249f909727fd452b"
server_key_iv: "856b7f55824a3d9c4a90b8459245f747"
server_key_version: "encryptionkey@0"
I20260923 17:22:24.200314 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "b8dddce61c854580bd646a4b4c4287c5"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "72a20d48a5013ec3249f909727fd452b"
server_key_iv: "856b7f55824a3d9c4a90b8459245f747"
server_key_version: "encryptionkey@0"
I20260923 17:22:24.204455 7597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:22:24.206910 7612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.208143 7597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:22:24.208254 7597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "b8dddce61c854580bd646a4b4c4287c5"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "72a20d48a5013ec3249f909727fd452b"
server_key_iv: "856b7f55824a3d9c4a90b8459245f747"
server_key_version: "encryptionkey@0"
I20260923 17:22:24.208348 7597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:24.222525 7597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:24.225795 7597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:24.226030 7597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:24.226568 7597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:24.227384 7597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:24.227432 7597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.227471 7597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:24.227494 7597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.236436 7597 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:33839
I20260923 17:22:24.236457 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:33839 every 8 connection(s)
I20260923 17:22:24.237323 7597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:24.243184 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7597
I20260923 17:22:24.243283 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:22:24.243572 24410 external_mini_cluster.cc:1428] Setting key 588827628f2b14e90eb5babd0dd76f01
Sep 23 17:22:24 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184144, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:24.251390 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.239068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:41175 (local address 127.23.214.190:34557)
0923 17:22:24.239212 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.239217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.240104 (+ 887us) server_negotiation.cc:407] Connection header received
0923 17:22:24.241401 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.241404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.241451 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.241524 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:24.242749 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.243350 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.244674 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.244997 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.245108 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.247649 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:24.247653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:24.247655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:24.247676 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:24.249055 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.249711 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.249714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.249715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.249809 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.250283 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.250285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.250286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.250459 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:24.250537 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.251174 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.251274 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:22:24.252664 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.239504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:41175)
0923 17:22:24.239952 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.239994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.241205 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.241686 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.241692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.242148 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:24.242626 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.242640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.243476 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.243484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.244522 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.244531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.245149 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.245151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.245288 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.245934 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:24.245950 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:24.247482 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:24.249204 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:24.249211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:24.249228 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:24.249597 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:24.249956 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:24.249963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:24.249968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:24.250167 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:24.250567 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:24.250576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:24.250923 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.251136 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.251489 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:22:24.253954 7728 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34557
I20260923 17:22:24.254200 7728 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:24.254753 7728 heartbeater.cc:511] Master 127.23.214.190:34557 requested a full tablet report, sending...
I20260923 17:22:24.255791 7398 ts_manager.cc:194] Registered new tserver with Master: b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
I20260923 17:22:24.256459 7398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:41175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:22:24.273777 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34557
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:24.385046 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:24.385354 7733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:24.385401 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:24.389662 7733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:24.389760 7733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:24.389876 7733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:22:24.395566 7733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:24.396692 7733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:24.397825 7733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:24.407843 7741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:24.407867 7733 server_base.cc:1034] running on GCE node
W20260923 17:22:24.407831 7739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:24.407827 7738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:24.408437 7733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:24.409138 7733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:24.410307 7733 hybrid_clock.cc:648] HybridClock initialized: now 1790184144410281 us; error 48 us; skew 500 ppm
Sep 23 17:22:24 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184144, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:24.413255 7733 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:22:24.414181 7733 webserver.cc:460] Webserver started at http://127.23.214.131:40773/ using document root <none> and password file <none>
I20260923 17:22:24.414739 7733 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:24.414830 7733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:24.415057 7733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:22:24.417119 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "f1d92582930c4a3f922cf686cb1a4e9d"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "5efc3ae9643495bffe3054b045d3c8fb"
server_key_iv: "bee776654cb98df0c7816814b8fa0411"
server_key_version: "encryptionkey@0"
I20260923 17:22:24.417682 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "f1d92582930c4a3f922cf686cb1a4e9d"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "5efc3ae9643495bffe3054b045d3c8fb"
server_key_iv: "bee776654cb98df0c7816814b8fa0411"
server_key_version: "encryptionkey@0"
I20260923 17:22:24.421850 7733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:22:24.424284 7749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.425529 7733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:24.425698 7733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "f1d92582930c4a3f922cf686cb1a4e9d"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "5efc3ae9643495bffe3054b045d3c8fb"
server_key_iv: "bee776654cb98df0c7816814b8fa0411"
server_key_version: "encryptionkey@0"
I20260923 17:22:24.425814 7733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:24.437203 7733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:24.440472 7733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:24.440696 7733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:24.441313 7733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:24.442225 7733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:24.442307 7733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.442382 7733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:24.442435 7733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:24.451838 7733 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:38535
I20260923 17:22:24.451854 7863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:38535 every 8 connection(s)
I20260923 17:22:24.452790 7733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:24.460024 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7733
I20260923 17:22:24.460175 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:22:24.460523 24410 external_mini_cluster.cc:1428] Setting key 74d610c34e1ebf95d41a7e9a6ff9e2d1
Sep 23 17:22:24 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184144, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:24.467067 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.454677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:55317 (local address 127.23.214.190:34557)
0923 17:22:24.454827 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.454831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.455613 (+ 782us) server_negotiation.cc:407] Connection header received
0923 17:22:24.456993 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.456996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.457047 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.457153 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:24.458536 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.459107 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.460317 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.460658 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.460767 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.463492 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:24.463496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:24.463498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:24.463523 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:24.464983 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.465487 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.465490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.465492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.465568 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.465995 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.465999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.466009 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.466225 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:24.466323 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.466834 (+ 511us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.466943 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:22:24.467962 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.454980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:55317)
0923 17:22:24.455437 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.455481 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.456786 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.457343 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.457350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.457836 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:24.458401 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.458415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.459226 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.459229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.460197 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.460208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.460802 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.460805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.461012 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.461761 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:24.461778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:24.463341 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:24.465105 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:24.465110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:24.465124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:24.465397 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:24.465658 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:24.465660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:24.465661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:24.465799 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:24.466343 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:24.466348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:24.466615 (+ 267us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.466784 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.467065 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260923 17:22:24.469233 7864 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34557
I20260923 17:22:24.469528 7864 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:24.470247 7864 heartbeater.cc:511] Master 127.23.214.190:34557 requested a full tablet report, sending...
I20260923 17:22:24.471499 7398 ts_manager.cc:194] Registered new tserver with Master: f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535)
I20260923 17:22:24.472119 7398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:55317
I20260923 17:22:24.474526 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:22:24.483522 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.476422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.23.214.190:34557)
0923 17:22:24.476604 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.476609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.476785 (+ 176us) server_negotiation.cc:407] Connection header received
0923 17:22:24.476883 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.476885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.476924 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.476989 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:24.477637 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.478228 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.479127 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.479334 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.479411 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.480672 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:24.480675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:24.480677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:24.480700 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:24.482097 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.482461 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.482463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.482465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.482534 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:24.482756 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:24.482758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:24.482760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:24.482960 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:24.483062 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.483243 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.483334 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:22:24.486588 7398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:24.489650 7398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:24.504441 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.498722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.23.214.131:38535)
0923 17:22:24.499788 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.499799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.499821 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:22:24.499893 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.499901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.500084 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.500236 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:24.500500 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.501333 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.503011 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.503774 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.503899 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.504045 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.504194 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.504290 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":171,"threads_started":1}
I20260923 17:22:24.504426 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.498324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:38535 (local address 127.0.0.1:49988)
0923 17:22:24.499069 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.499085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.499169 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.500237 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.500240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.500257 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:24.500376 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.500383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.501461 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.501467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.502891 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.502900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.503892 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.503895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.503981 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.504011 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.504155 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.504275 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260923 17:22:24.504709 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.498236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:38519 (local address 127.0.0.1:43854)
0923 17:22:24.498807 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.498859 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.499157 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.500017 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.500026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.500055 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:24.500233 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.500240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.501910 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.501913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.503223 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.503229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.504331 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.504335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.504424 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.504447 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.504493 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.504569 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":97,"thread_start_us":93,"threads_started":1}
I20260923 17:22:24.505565 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.498362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.23.214.129:38519)
0923 17:22:24.499472 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.499489 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.499514 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:22:24.499608 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.499617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.499840 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.500003 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:24.500387 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.501757 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.503366 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.504197 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.505107 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.505241 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.505292 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.505403 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":67,"threads_started":1}
I20260923 17:22:24.506392 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.498942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33839 (local address 127.0.0.1:58660)
0923 17:22:24.499395 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.499410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.499490 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.500138 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.500140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.500154 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:24.500241 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.500247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.501814 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.501821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.503218 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.503229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.505975 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.505979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.506095 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.506122 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.506206 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.506271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":118,"thread_start_us":83,"threads_started":1}
I20260923 17:22:24.506796 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.499342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.23.214.130:33839)
0923 17:22:24.499714 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.499722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.499740 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:22:24.499798 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.499802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.499973 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.500387 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:24.500541 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.501656 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.504123 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.504974 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.505269 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.505378 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.506254 (+ 876us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.506352 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":99,"threads_started":1}
I20260923 17:22:24.507758 7797 tablet_service.cc:1467] Processing CreateTablet for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf (DEFAULT_TABLE table=test-table [id=86df09c21cea4bc4a5c84eaa9eb51f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:24.508217 7525 tablet_service.cc:1467] Processing CreateTablet for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf (DEFAULT_TABLE table=test-table [id=86df09c21cea4bc4a5c84eaa9eb51f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:24.508944 7797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:24.509212 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:24.509325 7660 tablet_service.cc:1467] Processing CreateTablet for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf (DEFAULT_TABLE table=test-table [id=86df09c21cea4bc4a5c84eaa9eb51f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:24.510330 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:24.515611 7881 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Bootstrap starting.
I20260923 17:22:24.517076 7883 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Bootstrap starting.
I20260923 17:22:24.517184 7882 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Bootstrap starting.
I20260923 17:22:24.518385 7881 tablet_bootstrap.cc:654] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:24.519155 7882 tablet_bootstrap.cc:654] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:24.519707 7883 tablet_bootstrap.cc:654] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:24.520381 7882 log.cc:824] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:24.520443 7881 log.cc:824] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:24.520443 7883 log.cc:824] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:24.522535 7881 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: No bootstrap required, opened a new log
I20260923 17:22:24.522543 7882 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: No bootstrap required, opened a new log
I20260923 17:22:24.522589 7883 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: No bootstrap required, opened a new log
I20260923 17:22:24.522749 7883 ts_tablet_manager.cc:1397] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:22:24.522749 7882 ts_tablet_manager.cc:1397] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260923 17:22:24.522755 7881 ts_tablet_manager.cc:1397] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:22:24.526192 7881 raft_consensus.cc:348] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.526448 7881 raft_consensus.cc:374] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:24.526505 7881 raft_consensus.cc:729] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37c2506020546d7894d35db6be03f7e, State: Initialized, Role: FOLLOWER
I20260923 17:22:24.526592 7883 raft_consensus.cc:348] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.526880 7883 raft_consensus.cc:374] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:24.526934 7883 raft_consensus.cc:729] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dddce61c854580bd646a4b4c4287c5, State: Initialized, Role: FOLLOWER
I20260923 17:22:24.526852 7882 raft_consensus.cc:348] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.527100 7882 raft_consensus.cc:374] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:24.527006 7881 consensus_queue.cc:260] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.527168 7882 raft_consensus.cc:729] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d92582930c4a3f922cf686cb1a4e9d, State: Initialized, Role: FOLLOWER
I20260923 17:22:24.527410 7883 consensus_queue.cc:260] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.527513 7882 consensus_queue.cc:260] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.528105 7728 heartbeater.cc:503] Master 127.23.214.190:34557 was elected leader, sending a full tablet report...
I20260923 17:22:24.528402 7882 ts_tablet_manager.cc:1428] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:22:24.528402 7883 ts_tablet_manager.cc:1428] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:22:24.528707 7881 ts_tablet_manager.cc:1428] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:22:24.528761 7864 heartbeater.cc:503] Master 127.23.214.190:34557 was elected leader, sending a full tablet report...
I20260923 17:22:24.529064 7592 heartbeater.cc:503] Master 127.23.214.190:34557 was elected leader, sending a full tablet report...
I20260923 17:22:24.659982 7889 raft_consensus.cc:482] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:24.660287 7889 raft_consensus.cc:504] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.661551 7889 leader_election.cc:288] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
I20260923 17:22:24.665293 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.661876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:38519 (local address 127.23.214.131:47079)
0923 17:22:24.662069 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.662088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.662201 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.662530 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.662533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.662551 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:24.662675 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.662681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.663428 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.663432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.664210 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.664218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.664854 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.664857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.664963 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.664991 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.665086 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.665163 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:22:24.665293 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.661980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:47079 (local address 127.23.214.129:38519)
0923 17:22:24.662129 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.662135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.662154 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:22:24.662340 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.662347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.662409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.662492 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:24.662791 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.663302 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.664340 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.664732 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.664865 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.664919 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.665098 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.665166 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:22:24.665705 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.662217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33839 (local address 127.23.214.131:50505)
0923 17:22:24.662580 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.662592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:24.662729 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:24.662971 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:24.662974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:24.662989 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:24.663118 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.663123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.663838 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.663843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.664613 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:24.664620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.665270 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.665274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:24.665370 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.665398 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:22:24.665505 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:22:24.665573 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":148,"threads_started":1}
I20260923 17:22:24.665714 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.662372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:50505 (local address 127.23.214.130:33839)
0923 17:22:24.662519 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.662523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.662696 (+ 173us) server_negotiation.cc:407] Connection header received
0923 17:22:24.662822 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.662825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.662875 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.662959 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:24.663234 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.663725 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.664740 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.665150 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.665279 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.665325 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.665529 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.665599 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:22:24.666013 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70d4dbaa7ce4ed7a6cd0d67770670bf" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37c2506020546d7894d35db6be03f7e" is_pre_election: true
I20260923 17:22:24.666407 7546 raft_consensus.cc:2454] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 0.
I20260923 17:22:24.666811 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70d4dbaa7ce4ed7a6cd0d67770670bf" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8dddce61c854580bd646a4b4c4287c5" is_pre_election: true
I20260923 17:22:24.666918 7750 leader_election.cc:302] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b37c2506020546d7894d35db6be03f7e, f1d92582930c4a3f922cf686cb1a4e9d; no voters:
I20260923 17:22:24.667137 7680 raft_consensus.cc:2454] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 0.
I20260923 17:22:24.667246 7889 raft_consensus.cc:2781] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:24.667337 7889 raft_consensus.cc:482] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:24.667392 7889 raft_consensus.cc:3037] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:24.668437 7889 raft_consensus.cc:504] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.668745 7889 leader_election.cc:288] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 election: Requested vote from peers b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
I20260923 17:22:24.669130 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70d4dbaa7ce4ed7a6cd0d67770670bf" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37c2506020546d7894d35db6be03f7e"
I20260923 17:22:24.669229 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70d4dbaa7ce4ed7a6cd0d67770670bf" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8dddce61c854580bd646a4b4c4287c5"
I20260923 17:22:24.669282 7546 raft_consensus.cc:3037] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:24.669399 7680 raft_consensus.cc:3037] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:24.670435 7546 raft_consensus.cc:2454] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 1.
I20260923 17:22:24.670569 7680 raft_consensus.cc:2454] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 1.
I20260923 17:22:24.670783 7750 leader_election.cc:302] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b37c2506020546d7894d35db6be03f7e, f1d92582930c4a3f922cf686cb1a4e9d; no voters:
I20260923 17:22:24.670976 7889 raft_consensus.cc:2781] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:24.671263 7889 raft_consensus.cc:686] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 1 LEADER]: Becoming Leader. State: Replica: f1d92582930c4a3f922cf686cb1a4e9d, State: Running, Role: LEADER
I20260923 17:22:24.671598 7889 consensus_queue.cc:237] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:24.674832 7397 catalog_manager.cc:5495] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d reported cstate change: term changed from 0 to 1, leader changed from <none> to f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131). New cstate: current_term: 1 leader_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } health_report { overall_health: UNKNOWN } } }
I20260923 17:22:24.698272 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.694965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.23.214.131:38535)
0923 17:22:24.695079 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:24.695086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:24.695126 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:22:24.695198 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:24.695200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:24.695254 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:24.695333 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:24.695543 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.696041 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.696831 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:24.697034 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:24.697109 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:24.697363 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:24.697787 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:24.697927 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:24.698023 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 17:22:24.698073 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260923 17:22:24.703699 7865 tablet.cc:2369] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:22:24.704716 7546 raft_consensus.cc:1261] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Refusing update from remote peer f1d92582930c4a3f922cf686cb1a4e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:24.704711 7680 raft_consensus.cc:1261] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Refusing update from remote peer f1d92582930c4a3f922cf686cb1a4e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:24.705415 7889 consensus_queue.cc:1035] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:24.705670 7891 consensus_queue.cc:1035] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:24.713783 7895 mvcc.cc:205] Tried to move back new op lower bound from 7332594256702435328 to 7332594256577351680. Current Snapshot: MvccSnapshot[applied={T|T < 7332594256702435328}]
I20260923 17:22:24.714031 7896 mvcc.cc:205] Tried to move back new op lower bound from 7332594256702435328 to 7332594256577351680. Current Snapshot: MvccSnapshot[applied={T|T < 7332594256702435328}]
I20260923 17:22:24.721717 7398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:24.721941 7398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:24.724515 7398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Sending DeleteTablet for 3 replicas of tablet c70d4dbaa7ce4ed7a6cd0d67770670bf
I20260923 17:22:24.725327 7797 tablet_service.cc:1514] Processing DeleteTablet for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:49988
I20260923 17:22:24.725334 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:58660
I20260923 17:22:24.725375 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:43854
I20260923 17:22:24.725905 7906 tablet_replica.cc:330] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: stopping tablet replica
I20260923 17:22:24.726176 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7462
I20260923 17:22:24.726265 7906 raft_consensus.cc:2229] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:24.726616 7906 raft_consensus.cc:2258] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:24.729322 7906 ts_tablet_manager.cc:1905] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:24.730093 7904 tablet_replica.cc:330] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: stopping tablet replica
I20260923 17:22:24.730648 7904 raft_consensus.cc:2229] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:24.731170 7904 raft_consensus.cc:2258] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:24.732458 7906 ts_tablet_manager.cc:1918] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:24.732564 7906 log.cc:1196] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/c70d4dbaa7ce4ed7a6cd0d67770670bf
I20260923 17:22:24.733045 7904 ts_tablet_manager.cc:1905] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:24.733249 7906 ts_tablet_manager.cc:1939] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b8dddce61c854580bd646a4b4c4287c5: Deleting consensus metadata
I20260923 17:22:24.734658 7382 catalog_manager.cc:4841] TS b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839): tablet c70d4dbaa7ce4ed7a6cd0d67770670bf (table test-table [id=86df09c21cea4bc4a5c84eaa9eb51f90]) successfully deleted
W20260923 17:22:24.735637 7382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:38519 (error 108)
I20260923 17:22:24.736454 7904 ts_tablet_manager.cc:1918] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:24.736485 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7597
I20260923 17:22:24.738322 7904 log.cc:1196] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/c70d4dbaa7ce4ed7a6cd0d67770670bf
I20260923 17:22:24.738732 7904 ts_tablet_manager.cc:1939] T c70d4dbaa7ce4ed7a6cd0d67770670bf P f1d92582930c4a3f922cf686cb1a4e9d: Deleting consensus metadata
I20260923 17:22:24.739943 7385 catalog_manager.cc:4841] TS f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): tablet c70d4dbaa7ce4ed7a6cd0d67770670bf (table test-table [id=86df09c21cea4bc4a5c84eaa9eb51f90]) successfully deleted
I20260923 17:22:24.743980 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:24.743660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:38519 (local address 127.0.0.1:43862)
0923 17:22:24.743836 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:24.743900 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260923 17:22:24.744225 7382 catalog_manager.cc:4584] TS b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf: Network error: Client connection negotiation failed: client connection to 127.23.214.129:38519: connect: Connection refused (error 111)
I20260923 17:22:24.745347 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7733
I20260923 17:22:24.752661 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7366
I20260923 17:22:44.759593 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34557
--webserver_interface=127.23.214.190
--webserver_port=46221
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:34557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:22:44.870960 7908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:44.871236 7908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:44.871317 7908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:44.875628 7908 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:22:44.875698 7908 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:22:44.875718 7908 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:22:44.875735 7908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:22:44.875795 7908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:44.875845 7908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:44.875913 7908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:22:44.875958 7908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:22:44.881590 7908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:34557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:44.882907 7908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:44.884033 7908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:44.898278 7914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:44.898279 7915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:44.910760 7917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:44.910840 7908 server_base.cc:1034] running on GCE node
I20260923 17:22:44.911599 7908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:44.912928 7908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:44.914119 7908 hybrid_clock.cc:648] HybridClock initialized: now 1790184164914102 us; error 38 us; skew 500 ppm
Sep 23 17:22:44 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184164, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:44.926573 7908 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:22:44.927592 7908 webserver.cc:460] Webserver started at http://127.23.214.190:46221/ using document root <none> and password file <none>
I20260923 17:22:44.928169 7908 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:44.928242 7908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:44.932274 7908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:22:44.934599 7924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:44.936018 7908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:44.936154 7908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "48f387240dfc4745a8575390ac9394a6"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "bcdd600db2659ced1bfa2a4c2933c5d6"
server_key_iv: "34743d09aa8a3db516f509fd663f4746"
server_key_version: "encryptionkey@0"
I20260923 17:22:44.936765 7908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:44.950743 7908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:44.953655 7908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:44.953907 7908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:44.961409 7908 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:34557
I20260923 17:22:44.961726 7976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:34557 every 8 connection(s)
I20260923 17:22:44.962579 7908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:22:44.966931 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7908
I20260923 17:22:44.967432 7977 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:22:44.967447 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:38519
--local_ip_for_outbound_sockets=127.23.214.129
--tserver_master_addrs=127.23.214.190:34557
--webserver_port=36005
--webserver_interface=127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:22:44.969961 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Bootstrap starting.
I20260923 17:22:44.987758 7977 log.cc:824] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:44.996553 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:44.996958 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Bootstrap complete.
I20260923 17:22:45.001042 7977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:45.001451 7977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f387240dfc4745a8575390ac9394a6, State: Initialized, Role: FOLLOWER
I20260923 17:22:45.002043 7977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:45.002172 7977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:22:45.002266 7977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:22:45.002379 7977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:22:45.004349 7977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:45.004688 7977 leader_election.cc:302] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f387240dfc4745a8575390ac9394a6; no voters:
I20260923 17:22:45.005014 7977 leader_election.cc:288] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:22:45.005149 7981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:22:45.005429 7981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 LEADER]: Becoming Leader. State: Replica: 48f387240dfc4745a8575390ac9394a6, State: Running, Role: LEADER
I20260923 17:22:45.005770 7981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:22:45.006292 7977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:22:45.007192 7982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48f387240dfc4745a8575390ac9394a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } } }
I20260923 17:22:45.007316 7982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:45.008011 7983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f387240dfc4745a8575390ac9394a6. Latest consensus state: current_term: 2 leader_uuid: "48f387240dfc4745a8575390ac9394a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } } }
I20260923 17:22:45.008330 7983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:22:45.011773 7998 catalog_manager.cc:1259] Loaded cluster ID: 461b41e05c5548dcaa009e67d615fa70
I20260923 17:22:45.011843 7998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: loading cluster ID for follower catalog manager: success
I20260923 17:22:45.014509 7998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: acquiring CA information for follower catalog manager: success
I20260923 17:22:45.015977 7998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:22:45.016275 7991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:22:45.018555 7991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:22:45.019022 7991 catalog_manager.cc:1259] Loaded cluster ID: 461b41e05c5548dcaa009e67d615fa70
I20260923 17:22:45.019079 7991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:22:45.019747 7991 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:22:45.020206 7991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Loaded TSK: 0
I20260923 17:22:45.040931 7991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Generated new TSK 1
I20260923 17:22:45.041332 7991 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:22:45.088088 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:45.088320 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:45.088361 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:45.092350 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:45.092406 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:45.092490 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:22:45.098037 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:38519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=36005
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:45.099076 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:45.100045 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:45.110569 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:45.110566 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:45.110803 7979 server_base.cc:1034] running on GCE node
W20260923 17:22:45.111089 8009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:45.111497 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:45.112179 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:45.113364 7979 hybrid_clock.cc:648] HybridClock initialized: now 1790184165113333 us; error 48 us; skew 500 ppm
Sep 23 17:22:45 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184165, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:45.116453 7979 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:22:45.117429 7979 webserver.cc:460] Webserver started at http://127.23.214.129:36005/ using document root <none> and password file <none>
I20260923 17:22:45.118122 7979 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:45.118249 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:45.122562 7979 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:45.124856 8016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:45.126358 7979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:45.126502 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b37c2506020546d7894d35db6be03f7e"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "d3382f16e52273f4d52996414c836944"
server_key_iv: "45a2552f76a53cd61c5db172032a65da"
server_key_version: "encryptionkey@0"
I20260923 17:22:45.127101 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:45.139225 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:45.142745 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:45.142961 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:45.143590 7979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:45.144815 8023 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:22:45.149223 7979 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:22:45.149286 7979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 17:22:45.149358 7979 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:22:45.151198 7979 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:22:45.151260 7979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260923 17:22:45.151525 8023 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Bootstrap starting.
I20260923 17:22:45.163201 7979 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:38519
I20260923 17:22:45.163233 8134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:38519 every 8 connection(s)
I20260923 17:22:45.164477 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:22:45.165467 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 7979
I20260923 17:22:45.166038 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:33839
--local_ip_for_outbound_sockets=127.23.214.130
--tserver_master_addrs=127.23.214.190:34557
--webserver_port=45863
--webserver_interface=127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:22:45 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184165, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:45.175781 8023 log.cc:824] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:45.183069 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.166767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:33257)
0923 17:22:45.167894 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.167950 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.168762 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.169830 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.169838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.170307 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:45.170888 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.170908 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.172298 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.172301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.173153 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.173161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.173930 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.173934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.174134 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.174752 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:45.174769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:45.176448 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:45.179513 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.179519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.179532 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.179872 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.180340 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.180343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.180345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.180506 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.181241 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:45.181247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:45.181527 (+ 280us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.181734 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.182158 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":161,"threads_started":1}
I20260923 17:22:45.183076 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.166760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:33257 (local address 127.23.214.190:34557)
0923 17:22:45.167886 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.167905 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.168082 (+ 177us) server_negotiation.cc:407] Connection header received
0923 17:22:45.169172 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.169182 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.169463 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.169783 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.171077 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.172128 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.173280 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.173740 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.173954 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.176681 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:45.176693 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:45.176714 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:45.176760 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:45.179366 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.180009 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.180018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.180028 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.180221 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.180621 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.180628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.180633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.181104 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:45.181264 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.181901 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.182233 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":114,"threads_started":1}
I20260923 17:22:45.183929 8023 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:22:45.184331 8023 tablet_bootstrap.cc:492] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Bootstrap complete.
I20260923 17:22:45.185595 8023 ts_tablet_manager.cc:1397] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.004s
I20260923 17:22:45.186209 8135 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34557
I20260923 17:22:45.186463 8135 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:45.187011 8135 heartbeater.cc:511] Master 127.23.214.190:34557 requested a full tablet report, sending...
I20260923 17:22:45.188023 8023 raft_consensus.cc:348] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:45.188472 8023 raft_consensus.cc:729] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37c2506020546d7894d35db6be03f7e, State: Initialized, Role: FOLLOWER
I20260923 17:22:45.188979 8023 consensus_queue.cc:260] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } } peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } }
I20260923 17:22:45.189822 7939 ts_manager.cc:194] Registered new tserver with Master: b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519)
I20260923 17:22:45.189997 8023 ts_tablet_manager.cc:1428] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Time spent starting tablet: real 0.004s user 0.008s sys 0.000s
I20260923 17:22:45.193004 7939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:33257
I20260923 17:22:45.195785 8135 heartbeater.cc:503] Master 127.23.214.190:34557 was elected leader, sending a full tablet report...
Sep 23 17:22:45 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184164, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.129@KRBTEST.COM
I20260923 17:22:45.204833 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.192585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:38519 (local address 127.0.0.1:60708)
0923 17:22:45.192999 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.193043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.193173 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.196126 (+ 2953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.196132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.196474 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:45.196583 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.196589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.197704 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.197711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.198439 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.198449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.199068 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.199072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.199183 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.199759 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:45.199771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:45.201076 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:45.203056 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.203064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.203070 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.203305 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.203622 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.203627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.203629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.203721 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.204159 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:45.204165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:45.204294 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.204432 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.204651 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":118,"threads_started":1}
I20260923 17:22:45.204833 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.193347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.23.214.129:38519)
0923 17:22:45.193637 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.193651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.193674 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:22:45.193738 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.193743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.195960 (+ 2217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.196127 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.196703 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.197592 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.198580 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.198935 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.199083 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.201211 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:45.201220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:45.201232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:45.201263 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:45.202929 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.203424 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.203429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.203431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.203521 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.203812 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.203817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.203819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.204038 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:45.204186 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.204520 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.204655 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":121,"threads_started":1}
I20260923 17:22:45.205827 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet c70d4dbaa7ce4ed7a6cd0d67770670bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:60708
I20260923 17:22:45.206832 8144 tablet_replica.cc:330] stopping tablet replica
I20260923 17:22:45.207151 8144 raft_consensus.cc:2229] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:45.207456 8144 raft_consensus.cc:2258] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:45.208598 8144 ts_tablet_manager.cc:1905] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:45.213150 8144 ts_tablet_manager.cc:1918] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:45.213291 8144 log.cc:1196] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal/wals/c70d4dbaa7ce4ed7a6cd0d67770670bf
I20260923 17:22:45.213632 8144 ts_tablet_manager.cc:1939] T c70d4dbaa7ce4ed7a6cd0d67770670bf P b37c2506020546d7894d35db6be03f7e: Deleting consensus metadata
I20260923 17:22:45.215658 7925 catalog_manager.cc:4841] TS b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519): tablet c70d4dbaa7ce4ed7a6cd0d67770670bf (table test-table [id=86df09c21cea4bc4a5c84eaa9eb51f90]) successfully deleted
W20260923 17:22:45.292798 8139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:45.293071 8139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:45.293147 8139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:45.297245 8139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:45.297308 8139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:45.297401 8139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:22:45.302748 8139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:33839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=45863
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:45.303882 8139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:45.304942 8139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:45.315243 8152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:45.315438 8139 server_base.cc:1034] running on GCE node
W20260923 17:22:45.315240 8150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:45.315306 8149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:45.316023 8139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:45.316704 8139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:45.317929 8139 hybrid_clock.cc:648] HybridClock initialized: now 1790184165317911 us; error 28 us; skew 500 ppm
Sep 23 17:22:45 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184165, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:45.320797 8139 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:22:45.321718 8139 webserver.cc:460] Webserver started at http://127.23.214.130:45863/ using document root <none> and password file <none>
I20260923 17:22:45.322352 8139 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:45.322461 8139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:45.326650 8139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:45.328895 8159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:45.330607 8139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:22:45.330797 8139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "b8dddce61c854580bd646a4b4c4287c5"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "72a20d48a5013ec3249f909727fd452b"
server_key_iv: "856b7f55824a3d9c4a90b8459245f747"
server_key_version: "encryptionkey@0"
I20260923 17:22:45.331462 8139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:45.406296 8139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:45.409634 8139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:45.409868 8139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:45.410535 8139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:45.411387 8139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:45.411469 8139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:45.411547 8139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:45.411590 8139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:45.420796 8139 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:33839
I20260923 17:22:45.420819 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:33839 every 8 connection(s)
I20260923 17:22:45.421805 8139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:22:45.426028 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 8139
I20260923 17:22:45.426537 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:38535
--local_ip_for_outbound_sockets=127.23.214.131
--tserver_master_addrs=127.23.214.190:34557
--webserver_port=40773
--webserver_interface=127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:22:45 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184165, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:45.438232 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.423728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:59939 (local address 127.23.214.190:34557)
0923 17:22:45.423865 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.423868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.425209 (+ 1341us) server_negotiation.cc:407] Connection header received
0923 17:22:45.426393 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.426396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.426449 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.426521 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.428255 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.429125 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.430468 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.430963 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.431108 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.434152 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:45.434156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:45.434158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:45.434184 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:45.435755 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.436459 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.436462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.436463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.436539 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.436992 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.436994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.436995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.437184 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:45.437261 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.437996 (+ 735us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.438107 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:22:45.439420 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.424135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:59939)
0923 17:22:45.425022 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.425068 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.426116 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.426769 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.426782 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.427337 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:45.428054 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.428078 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.429269 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.429271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.430352 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.430361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.431160 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.431165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.431314 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.431953 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:45.431970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:45.433958 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:45.435899 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.435907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.435926 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.436341 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.436647 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.436653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.436657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.436882 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.437297 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:45.437306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:45.437689 (+ 383us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.437946 (+ 257us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.438360 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":536,"threads_started":1}
I20260923 17:22:45.440729 8274 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34557
I20260923 17:22:45.440997 8274 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:45.441546 8274 heartbeater.cc:511] Master 127.23.214.190:34557 requested a full tablet report, sending...
I20260923 17:22:45.442656 7939 ts_manager.cc:194] Registered new tserver with Master: b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
I20260923 17:22:45.443244 7939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:59939
W20260923 17:22:45.541230 8277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:22:45.541566 8277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:22:45.541675 8277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:22:45.545734 8277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:22:45.545851 8277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:22:45.546113 8277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:22:45.551723 8277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:38535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=40773
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:22:45.552801 8277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:22:45.553849 8277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:22:45.563645 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:45.563699 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:22:45.563645 8283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:22:45.563932 8277 server_base.cc:1034] running on GCE node
I20260923 17:22:45.564401 8277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:22:45.565075 8277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:22:45.566258 8277 hybrid_clock.cc:648] HybridClock initialized: now 1790184165566244 us; error 28 us; skew 500 ppm
Sep 23 17:22:45 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184165, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:22:45.568935 8277 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:22:45.569782 8277 webserver.cc:460] Webserver started at http://127.23.214.131:40773/ using document root <none> and password file <none>
I20260923 17:22:45.570380 8277 fs_manager.cc:359] Metadata directory not provided
I20260923 17:22:45.570484 8277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:22:45.574259 8277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 17:22:45.576412 8292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:45.577643 8277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:22:45.577785 8277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "f1d92582930c4a3f922cf686cb1a4e9d"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "5efc3ae9643495bffe3054b045d3c8fb"
server_key_iv: "bee776654cb98df0c7816814b8fa0411"
server_key_version: "encryptionkey@0"
I20260923 17:22:45.578379 8277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:22:45.601591 8277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:22:45.604699 8277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:22:45.604921 8277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:22:45.605535 8277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:22:45.606473 8277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:22:45.606546 8277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:45.606618 8277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:22:45.606655 8277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:22:45.617067 8277 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:38535
I20260923 17:22:45.617089 8409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:38535 every 8 connection(s)
I20260923 17:22:45.618163 8277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:22:45.624156 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 8277
Sep 23 17:22:45 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184165, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:22:45.632745 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.620067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:37239 (local address 127.23.214.190:34557)
0923 17:22:45.620227 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.620231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.621064 (+ 833us) server_negotiation.cc:407] Connection header received
0923 17:22:45.622480 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.622483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.622531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.622603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.623782 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.624550 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.625921 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.626312 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.626424 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.628995 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:45.628998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:45.629001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:45.629021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:45.630479 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.631002 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.631006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.631008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.631086 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.631505 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.631510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.631512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.631766 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:45.631855 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.632470 (+ 615us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.632599 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:22:45.633589 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.624862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.23.214.190:34557)
0923 17:22:45.625103 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.625109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.625337 (+ 228us) server_negotiation.cc:407] Connection header received
0923 17:22:45.625495 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.625498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.625548 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.625678 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.626585 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.627322 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.628206 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.628433 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.628547 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.630194 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:45.630197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:45.630199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:45.630231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:45.631807 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.632232 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.632236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.632237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.632318 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.632599 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.632603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.632604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.632804 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:45.633003 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.633342 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.633452 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260923 17:22:45.633795 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.620472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:37239)
0923 17:22:45.620910 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.620952 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.622279 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.622771 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.622778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.623151 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:22:45.623646 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.623659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.624681 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.624687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.625783 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.625793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.626477 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.626480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.626711 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.627322 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:22:45.627347 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:22:45.628825 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:22:45.630589 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.630595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.630613 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.630896 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.631184 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:22:45.631187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:22:45.631189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:22:45.631376 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:22:45.631886 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:22:45.631895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:22:45.632157 (+ 262us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.632398 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.632760 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260923 17:22:45.635192 8410 heartbeater.cc:347] Connected to a master server at 127.23.214.190:34557
I20260923 17:22:45.635445 8410 heartbeater.cc:464] Registering TS with master...
I20260923 17:22:45.635985 8410 heartbeater.cc:511] Master 127.23.214.190:34557 requested a full tablet report, sending...
I20260923 17:22:45.637036 7939 ts_manager.cc:194] Registered new tserver with Master: f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535)
I20260923 17:22:45.637583 7939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:37239
I20260923 17:22:45.647405 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:22:45.651381 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.648098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.23.214.190:34557)
0923 17:22:45.648221 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.648241 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.648417 (+ 176us) server_negotiation.cc:407] Connection header received
0923 17:22:45.648569 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.648571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.648674 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.648766 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:45.648972 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.649488 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.650388 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.650603 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.650682 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.650943 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:45.651073 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 17:22:45.651282 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44960: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 17:22:45.651467 8413 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44960: token expired
W20260923 17:22:45.651755 7870 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.214.190:34557)
I20260923 17:22:45.658483 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.652105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.23.214.190:34557)
0923 17:22:45.652209 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.652212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.652223 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:22:45.652316 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.652317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.652351 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.652413 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:22:45.653012 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.653465 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.654291 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.654485 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.654560 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.655834 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:22:45.655837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:22:45.655839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:22:45.655855 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:22:45.657117 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.657490 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.657493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.657494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.657565 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:22:45.657785 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:22:45.657787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:22:45.657788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:22:45.657997 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:22:45.658097 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.658285 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.658373 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:22:45.660686 7939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:22:45.663966 7939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:22:45.672116 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 01c8b5659e634f0b9441c9492abd9f19 (DEFAULT_TABLE table=test-table [id=b146bc56e2f746478ce739ceeb8a5c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:45.672855 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c8b5659e634f0b9441c9492abd9f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:45.675746 8023 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Bootstrap starting.
I20260923 17:22:45.676661 8023 tablet_bootstrap.cc:654] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:45.676944 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.671103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33839 (local address 127.0.0.1:34430)
0923 17:22:45.671402 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.671417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.671503 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.672227 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.672230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.672260 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:45.672513 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.672521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.674142 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.674147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.675047 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.675056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.676421 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.676425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.676534 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.676598 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.676695 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.676777 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":203,"spinlock_wait_cycles":896,"thread_start_us":128,"threads_started":1}
I20260923 17:22:45.676968 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.671293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.23.214.130:33839)
0923 17:22:45.671762 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.671775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.671798 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:22:45.671872 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.671877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.672063 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.672349 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:45.672637 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.674027 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.675179 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.676289 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.676423 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.676563 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.676728 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.676824 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":121,"threads_started":1}
I20260923 17:22:45.676980 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.670827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:38535 (local address 127.0.0.1:56894)
0923 17:22:45.670999 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.671011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.671103 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.672177 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.672180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.672197 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:45.672352 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.672362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.674049 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.674052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.675339 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.675350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.676442 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.676446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.676633 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.676662 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.676744 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.676831 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":89,"spinlock_wait_cycles":3456}
I20260923 17:22:45.677706 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.671358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.23.214.131:38535)
0923 17:22:45.671766 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.671775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.671795 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:22:45.671859 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.671863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.672031 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.672199 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:45.672484 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.673922 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.675494 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.676304 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.677248 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.677393 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.677447 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.677562 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260923 17:22:45.678385 8023 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: No bootstrap required, opened a new log
I20260923 17:22:45.678485 8023 ts_tablet_manager.cc:1397] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:22:45.678936 8023 raft_consensus.cc:348] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.679064 8023 raft_consensus.cc:374] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:45.679103 8023 raft_consensus.cc:729] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37c2506020546d7894d35db6be03f7e, State: Initialized, Role: FOLLOWER
I20260923 17:22:45.679242 8023 consensus_queue.cc:260] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.679513 8023 ts_tablet_manager.cc:1428] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 17:22:45.680169 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 01c8b5659e634f0b9441c9492abd9f19 (DEFAULT_TABLE table=test-table [id=b146bc56e2f746478ce739ceeb8a5c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:45.680536 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 01c8b5659e634f0b9441c9492abd9f19 (DEFAULT_TABLE table=test-table [id=b146bc56e2f746478ce739ceeb8a5c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:22:45.681386 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c8b5659e634f0b9441c9492abd9f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:45.681591 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c8b5659e634f0b9441c9492abd9f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:22:45.688365 8423 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Bootstrap starting.
I20260923 17:22:45.688827 8424 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Bootstrap starting.
I20260923 17:22:45.690855 8424 tablet_bootstrap.cc:654] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:45.690982 8423 tablet_bootstrap.cc:654] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Neither blocks nor log segments found. Creating new log.
I20260923 17:22:45.691731 8423 log.cc:824] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:45.691942 8424 log.cc:824] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:22:45.693480 8423 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: No bootstrap required, opened a new log
I20260923 17:22:45.693563 8424 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: No bootstrap required, opened a new log
I20260923 17:22:45.693725 8423 ts_tablet_manager.cc:1397] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:22:45.693768 8424 ts_tablet_manager.cc:1397] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:22:45.697064 8423 raft_consensus.cc:348] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.697065 8424 raft_consensus.cc:348] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.697353 8423 raft_consensus.cc:374] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:45.697348 8424 raft_consensus.cc:374] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:22:45.697422 8424 raft_consensus.cc:729] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d92582930c4a3f922cf686cb1a4e9d, State: Initialized, Role: FOLLOWER
I20260923 17:22:45.697425 8423 raft_consensus.cc:729] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dddce61c854580bd646a4b4c4287c5, State: Initialized, Role: FOLLOWER
I20260923 17:22:45.697935 8423 consensus_queue.cc:260] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.698077 8424 consensus_queue.cc:260] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.698768 8274 heartbeater.cc:503] Master 127.23.214.190:34557 was elected leader, sending a full tablet report...
I20260923 17:22:45.698825 8410 heartbeater.cc:503] Master 127.23.214.190:34557 was elected leader, sending a full tablet report...
I20260923 17:22:45.699152 8423 ts_tablet_manager.cc:1428] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:22:45.699152 8424 ts_tablet_manager.cc:1428] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:22:45.738492 8428 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:22:45.738824 8428 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.740345 8428 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839), b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519)
I20260923 17:22:45.744338 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.740719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:59743 (local address 127.23.214.130:33839)
0923 17:22:45.740876 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.740882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.740900 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:22:45.740966 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.740972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.741038 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.741130 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:45.741457 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.742185 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.743246 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.743695 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.743849 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.743922 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.744142 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.744216 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:22:45.744338 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.740534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33839 (local address 127.23.214.131:59743)
0923 17:22:45.740703 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.740718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.740827 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.741139 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.741142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.741159 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:45.741326 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.741334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.742297 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.742303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.743103 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.743112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.743837 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.743841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.743966 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.743996 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.744112 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.744188 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:22:45.744509 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.741055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:52811 (local address 127.23.214.129:38519)
0923 17:22:45.741388 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.741395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.741409 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:22:45.741454 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.741457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.741514 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.741633 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:22:45.741969 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.742461 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.743459 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.743941 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.744068 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.744240 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.744282 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.744348 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":83,"threads_started":1}
I20260923 17:22:45.744823 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.740813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:38519 (local address 127.23.214.131:52811)
0923 17:22:45.741187 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.741236 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:22:45.741328 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:22:45.741667 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:22:45.741670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:22:45.741690 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:22:45.741817 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.741824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.742569 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.742576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.743331 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:22:45.743340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.744057 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.744061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:22:45.744171 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.744192 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 17:22:45.744267 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:22:45.744332 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":133,"threads_started":1}
I20260923 17:22:45.745111 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c8b5659e634f0b9441c9492abd9f19" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8dddce61c854580bd646a4b4c4287c5" is_pre_election: true
I20260923 17:22:45.745481 8228 raft_consensus.cc:2454] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 0.
I20260923 17:22:45.745468 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c8b5659e634f0b9441c9492abd9f19" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37c2506020546d7894d35db6be03f7e" is_pre_election: true
I20260923 17:22:45.745716 8089 raft_consensus.cc:2454] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 0.
I20260923 17:22:45.746025 8293 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d; no voters:
I20260923 17:22:45.746271 8428 raft_consensus.cc:2781] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:22:45.746377 8428 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:22:45.746445 8428 raft_consensus.cc:3037] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:45.747622 8428 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.748014 8428 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 election: Requested vote from peers b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839), b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519)
I20260923 17:22:45.748576 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c8b5659e634f0b9441c9492abd9f19" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8dddce61c854580bd646a4b4c4287c5"
I20260923 17:22:45.748598 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c8b5659e634f0b9441c9492abd9f19" candidate_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37c2506020546d7894d35db6be03f7e"
I20260923 17:22:45.748755 8089 raft_consensus.cc:3037] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:45.748749 8228 raft_consensus.cc:3037] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:22:45.750075 8228 raft_consensus.cc:2454] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 1.
I20260923 17:22:45.750075 8089 raft_consensus.cc:2454] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d92582930c4a3f922cf686cb1a4e9d in term 1.
I20260923 17:22:45.750458 8293 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d; no voters:
I20260923 17:22:45.750655 8428 raft_consensus.cc:2781] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:22:45.750931 8428 raft_consensus.cc:686] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 1 LEADER]: Becoming Leader. State: Replica: f1d92582930c4a3f922cf686cb1a4e9d, State: Running, Role: LEADER
I20260923 17:22:45.751292 8428 consensus_queue.cc:237] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:22:45.754704 7939 catalog_manager.cc:5495] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d reported cstate change: term changed from 0 to 1, leader changed from <none> to f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131). New cstate: current_term: 1 leader_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } health_report { overall_health: UNKNOWN } } }
I20260923 17:22:45.783318 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.779700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.23.214.131:38535)
0923 17:22:45.779861 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:22:45.779868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:22:45.779881 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:22:45.779959 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:22:45.779961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:22:45.780010 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:22:45.780121 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:22:45.780363 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.780917 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.781762 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:22:45.782050 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:22:45.782159 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:22:45.782485 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:22:45.782896 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:22:45.783032 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:22:45.783092 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 17:22:45.783148 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:22:45.789423 8089 raft_consensus.cc:1261] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Refusing update from remote peer f1d92582930c4a3f922cf686cb1a4e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:45.789424 8228 raft_consensus.cc:1261] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Refusing update from remote peer f1d92582930c4a3f922cf686cb1a4e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:22:45.790199 8431 consensus_queue.cc:1035] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:45.790417 8428 consensus_queue.cc:1035] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:22:45.799775 8435 mvcc.cc:205] Tried to move back new op lower bound from 7332594343065251840 to 7332594342919950336. Current Snapshot: MvccSnapshot[applied={T|T < 7332594343065251840 or (T in {7332594343065251840})}]
I20260923 17:22:45.805361 7940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:45.805545 7940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:22:45.807993 7940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Sending DeleteTablet for 3 replicas of tablet 01c8b5659e634f0b9441c9492abd9f19
I20260923 17:22:45.808890 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c8b5659e634f0b9441c9492abd9f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:45 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:34430
I20260923 17:22:45.808890 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c8b5659e634f0b9441c9492abd9f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:22:45 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:56894
I20260923 17:22:45.809079 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7979
I20260923 17:22:45.809976 8443 tablet_replica.cc:330] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: stopping tablet replica
I20260923 17:22:45.810367 8443 raft_consensus.cc:2229] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:22:45.810869 8443 raft_consensus.cc:2258] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:45.811245 8444 tablet_replica.cc:330] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: stopping tablet replica
I20260923 17:22:45.811555 8444 raft_consensus.cc:2229] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:22:45.811923 8444 raft_consensus.cc:2258] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:22:45.813338 8443 ts_tablet_manager.cc:1905] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:45.813755 8444 ts_tablet_manager.cc:1905] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:22:45.816460 8444 ts_tablet_manager.cc:1918] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:22:45.816629 8444 log.cc:1196] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/01c8b5659e634f0b9441c9492abd9f19
I20260923 17:22:45.817039 8444 ts_tablet_manager.cc:1939] T 01c8b5659e634f0b9441c9492abd9f19 P b8dddce61c854580bd646a4b4c4287c5: Deleting consensus metadata
I20260923 17:22:45.816852 8443 ts_tablet_manager.cc:1918] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260923 17:22:45.817063 7925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:38519 (error 108)
I20260923 17:22:45.817162 8443 log.cc:1196] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/01c8b5659e634f0b9441c9492abd9f19
I20260923 17:22:45.817503 8443 ts_tablet_manager.cc:1939] T 01c8b5659e634f0b9441c9492abd9f19 P f1d92582930c4a3f922cf686cb1a4e9d: Deleting consensus metadata
I20260923 17:22:45.818295 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 8139
I20260923 17:22:45.818666 7928 catalog_manager.cc:4841] TS f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): tablet 01c8b5659e634f0b9441c9492abd9f19 (table test-table [id=b146bc56e2f746478ce739ceeb8a5c84]) successfully deleted
I20260923 17:22:45.821012 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.820697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:38519 (local address 127.0.0.1:60716)
0923 17:22:45.820834 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.820914 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:22:45.821171 7925 catalog_manager.cc:4584] TS b37c2506020546d7894d35db6be03f7e (127.23.214.129:38519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01c8b5659e634f0b9441c9492abd9f19: Network error: Client connection negotiation failed: client connection to 127.23.214.129:38519: connect: Connection refused (error 111)
I20260923 17:22:45.825834 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 8277
I20260923 17:22:45.827709 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:22:45.825961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33839 (local address 127.0.0.1:34436)
0923 17:22:45.827543 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0923 17:22:45.827608 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.130:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1475}
I20260923 17:22:45.831776 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 7908
I20260923 17:23:05.838771 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34557
--webserver_interface=127.23.214.190
--webserver_port=46221
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:34557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:05.954800 8446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:05.955111 8446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:05.955156 8446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:05.959317 8446 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:23:05.959393 8446 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:23:05.959411 8446 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:23:05.959427 8446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:05.959450 8446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:05.959522 8446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:05.959566 8446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:05.959609 8446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:05.965312 8446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:34557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:05.966503 8446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:05.967617 8446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:23:05.976972 8446 server_base.cc:1034] running on GCE node
W20260923 17:23:05.976954 8452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:05.977015 8451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:05.976997 8454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:05.977788 8446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:05.978976 8446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:05.980141 8446 hybrid_clock.cc:648] HybridClock initialized: now 1790184185980130 us; error 26 us; skew 500 ppm
Sep 23 17:23:05 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184185, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:05.983106 8446 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:05.984097 8446 webserver.cc:460] Webserver started at http://127.23.214.190:46221/ using document root <none> and password file <none>
I20260923 17:23:05.984675 8446 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:05.984738 8446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:05.988848 8446 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:23:05.991273 8461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:05.992624 8446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:23:05.992763 8446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "48f387240dfc4745a8575390ac9394a6"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "bcdd600db2659ced1bfa2a4c2933c5d6"
server_key_iv: "34743d09aa8a3db516f509fd663f4746"
server_key_version: "encryptionkey@0"
I20260923 17:23:05.993338 8446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:06.024310 8446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:06.027797 8446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:06.028021 8446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:06.035645 8446 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:34557
I20260923 17:23:06.035943 8514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:34557 every 8 connection(s)
I20260923 17:23:06.036866 8446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:06.041282 8515 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:23:06.043061 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Bootstrap starting.
I20260923 17:23:06.046453 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 8446
I20260923 17:23:06.046957 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:38519
--local_ip_for_outbound_sockets=127.23.214.129
--tserver_master_addrs=127.23.214.190:34557
--webserver_port=36005
--webserver_interface=127.23.214.129
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:23:06.061977 8515 log.cc:824] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:06.073415 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:23:06.073855 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: Bootstrap complete.
I20260923 17:23:06.078238 8515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:23:06.078657 8515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f387240dfc4745a8575390ac9394a6, State: Initialized, Role: FOLLOWER
I20260923 17:23:06.079208 8515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:23:06.079325 8515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:06.079406 8515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:06.079530 8515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:23:06.081642 8515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:23:06.082038 8515 leader_election.cc:302] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f387240dfc4745a8575390ac9394a6; no voters:
I20260923 17:23:06.082407 8515 leader_election.cc:288] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:23:06.082471 8519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:23:06.082723 8519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [term 3 LEADER]: Becoming Leader. State: Replica: 48f387240dfc4745a8575390ac9394a6, State: Running, Role: LEADER
I20260923 17:23:06.083101 8519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } }
I20260923 17:23:06.083743 8515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:06.084681 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f387240dfc4745a8575390ac9394a6. Latest consensus state: current_term: 3 leader_uuid: "48f387240dfc4745a8575390ac9394a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } } }
I20260923 17:23:06.084723 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "48f387240dfc4745a8575390ac9394a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f387240dfc4745a8575390ac9394a6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 34557 } } }
I20260923 17:23:06.084815 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:06.084847 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:06.088601 8534 catalog_manager.cc:1259] Loaded cluster ID: 461b41e05c5548dcaa009e67d615fa70
I20260923 17:23:06.088691 8534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: loading cluster ID for follower catalog manager: success
I20260923 17:23:06.090692 8534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48f387240dfc4745a8575390ac9394a6: acquiring CA information for follower catalog manager: success
F20260923 17:23:06.091955 8534 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe92d1b19ad google::LogMessage::Fail() at ??:0
@ 0x7fe92d1b488a google::LogMessage::SendToLog() at ??:0
@ 0x7fe92d1b14de google::LogMessage::Flush() at ??:0
@ 0x7fe92d1b2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe92fe7518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe937b16989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fe937af3023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fe937af354b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fe937aec652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fe937aeb9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fe937b32541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe92f3f4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe930a386db start_thread at ??:0
@ 0x7fe92bcaf61f clone at ??:0
W20260923 17:23:06.178359 8517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:06.178637 8517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:06.178683 8517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:06.182827 8517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:06.182886 8517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:06.182945 8517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:23:06.188328 8517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:38519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=36005
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:06.189481 8517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:06.190611 8517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:06.200529 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:06.200593 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:06.200732 8517 server_base.cc:1034] running on GCE node
W20260923 17:23:06.200549 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:06.201489 8517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:06.202322 8517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:06.203516 8517 hybrid_clock.cc:648] HybridClock initialized: now 1790184186203475 us; error 61 us; skew 500 ppm
Sep 23 17:23:06 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184186, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:06.206966 8517 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:23:06.208000 8517 webserver.cc:460] Webserver started at http://127.23.214.129:36005/ using document root <none> and password file <none>
I20260923 17:23:06.208621 8517 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:06.208729 8517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:06.213104 8517 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:23:06.215559 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:06.216940 8517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:23:06.217095 8517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "b37c2506020546d7894d35db6be03f7e"
format_stamp: "Formatted at 2026-09-23 17:22:23 on dist-test-slave-9q7f"
server_key: "d3382f16e52273f4d52996414c836944"
server_key_iv: "45a2552f76a53cd61c5db172032a65da"
server_key_version: "encryptionkey@0"
I20260923 17:23:06.217680 8517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:06.247272 8517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:06.250931 8517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:06.251166 8517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:06.251843 8517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:06.253120 8572 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 17:23:06.259198 8517 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:23:06.259332 8517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 17:23:06.259429 8517 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:23:06.261312 8517 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:23:06.261405 8517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 17:23:06.261747 8572 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Bootstrap starting.
I20260923 17:23:06.272778 8517 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:38519
I20260923 17:23:06.272866 8681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:38519 every 8 connection(s)
I20260923 17:23:06.273783 8517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:06.274684 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 8517
I20260923 17:23:06.275188 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:33839
--local_ip_for_outbound_sockets=127.23.214.130
--tserver_master_addrs=127.23.214.190:34557
--webserver_port=45863
--webserver_interface=127.23.214.130
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:23:06.284045 8572 log.cc:824] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:06.291420 8572 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:23:06.291848 8572 tablet_bootstrap.cc:492] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Bootstrap complete.
I20260923 17:23:06.292968 8572 ts_tablet_manager.cc:1397] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.014s
I20260923 17:23:06.295454 8572 raft_consensus.cc:348] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:06.295895 8572 raft_consensus.cc:729] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37c2506020546d7894d35db6be03f7e, State: Initialized, Role: FOLLOWER
I20260923 17:23:06.296384 8572 consensus_queue.cc:260] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:06.297356 8572 ts_tablet_manager.cc:1428] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260923 17:23:06.326036 8685 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:23:06.276479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:47681)
0923 17:23:06.277348 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.277408 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:06.278692 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:06.325211 (+ 46519us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":164,"threads_started":1}
W20260923 17:23:06.326656 8682 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:34557 (0 consecutive failures): Network error: Failed to ping master at 127.23.214.190:34557: Client connection negotiation failed: client connection to 127.23.214.190:34557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:23:06.327395 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.327133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:44451)
0923 17:23:06.327260 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.327329 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 17:23:06.328117 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.327902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:38101)
0923 17:23:06.328011 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.328058 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:23:06.328370 8682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:23:06.404774 8684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:06.405016 8684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:06.405059 8684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:06.408995 8684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:06.409046 8684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:06.409108 8684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:23:06.414315 8684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:33839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=45863
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:06.415305 8684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:06.416246 8684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:06.425840 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:06.425931 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:06.426131 8684 server_base.cc:1034] running on GCE node
W20260923 17:23:06.426317 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:06.426704 8684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:06.427367 8684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:06.428580 8684 hybrid_clock.cc:648] HybridClock initialized: now 1790184186428545 us; error 45 us; skew 500 ppm
Sep 23 17:23:06 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184186, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:06.431624 8684 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:23:06.432551 8684 webserver.cc:460] Webserver started at http://127.23.214.130:45863/ using document root <none> and password file <none>
I20260923 17:23:06.433153 8684 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:06.433259 8684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:06.437143 8684 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:23:06.439445 8704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:06.440737 8684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:06.440913 8684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "b8dddce61c854580bd646a4b4c4287c5"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "72a20d48a5013ec3249f909727fd452b"
server_key_iv: "856b7f55824a3d9c4a90b8459245f747"
server_key_version: "encryptionkey@0"
I20260923 17:23:06.441474 8684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:06.453871 8684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:06.457221 8684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:06.457435 8684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:06.458114 8684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:06.458953 8684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:06.459024 8684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:06.459100 8684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:06.459136 8684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:06.468529 8684 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:33839
I20260923 17:23:06.468611 8823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:33839 every 8 connection(s)
I20260923 17:23:06.469604 8684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:06.473727 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.471958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:38251)
0923 17:23:06.472434 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.472546 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
I20260923 17:23:06.474074 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 8684
W20260923 17:23:06.474246 8824 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:34557 (0 consecutive failures): Network error: Failed to ping master at 127.23.214.190:34557: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
I20260923 17:23:06.474620 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:38535
--local_ip_for_outbound_sockets=127.23.214.131
--tserver_master_addrs=127.23.214.190:34557
--webserver_port=40773
--webserver_interface=127.23.214.131
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:23:06.474965 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.474671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:36425)
0923 17:23:06.474842 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.474894 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:23:06.476758 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.476530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:49591)
0923 17:23:06.476663 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.476708 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 17:23:06.476955 8824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:23:06.586627 8827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:06.586921 8827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:06.586997 8827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:06.591068 8827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:06.591150 8827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:06.591234 8827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:23:06.596557 8827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:38535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=40773
--tserver_master_addrs=127.23.214.190:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:06.597612 8827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:06.598692 8827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:06.608237 8836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:06.608283 8833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:06.608250 8834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:06.608300 8827 server_base.cc:1034] running on GCE node
I20260923 17:23:06.608827 8827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:06.609434 8827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:06.610587 8827 hybrid_clock.cc:648] HybridClock initialized: now 1790184186610555 us; error 48 us; skew 500 ppm
Sep 23 17:23:06 dist-test-slave-9q7f krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184186, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:06.613189 8827 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:23:06.614110 8827 webserver.cc:460] Webserver started at http://127.23.214.131:40773/ using document root <none> and password file <none>
I20260923 17:23:06.614660 8827 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:06.614727 8827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:06.618577 8827 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:23:06.620675 8843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:06.622046 8827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:23:06.622160 8827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "f1d92582930c4a3f922cf686cb1a4e9d"
format_stamp: "Formatted at 2026-09-23 17:22:24 on dist-test-slave-9q7f"
server_key: "5efc3ae9643495bffe3054b045d3c8fb"
server_key_iv: "bee776654cb98df0c7816814b8fa0411"
server_key_version: "encryptionkey@0"
I20260923 17:23:06.622772 8827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:06.637356 8827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:06.640549 8827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:06.640780 8827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:06.641417 8827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:06.642302 8827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:06.642349 8827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:06.642410 8827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:06.642443 8827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:06.651821 8827 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:38535
I20260923 17:23:06.651861 8962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:38535 every 8 connection(s)
I20260923 17:23:06.652868 8827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:06.656821 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.655125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:53009)
0923 17:23:06.655582 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.655697 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
W20260923 17:23:06.657258 8963 heartbeater.cc:644] Failed to heartbeat to 127.23.214.190:34557 (0 consecutive failures): Network error: Failed to ping master at 127.23.214.190:34557: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
I20260923 17:23:06.657821 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.657600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:57967)
0923 17:23:06.657715 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.657762 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:23:06.658483 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:06.658273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:43917)
0923 17:23:06.658384 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:06.658427 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:23:06.658631 8963 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:23:06.661755 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 8827
I20260923 17:23:07.329375 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:07.328944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:33621)
0923 17:23:07.329213 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:07.329274 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260923 17:23:07.478145 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:07.477636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:49653)
0923 17:23:07.477972 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:07.478034 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260923 17:23:07.659577 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:07.659166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:44289)
0923 17:23:07.659418 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:07.659476 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260923 17:23:08.008034 8970 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:08.008265 8970 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:08.009338 8970 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
Sep 23 17:23:08 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184186, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.130@KRBTEST.COM
Sep 23 17:23:08 dist-test-slave-9q7f krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184186, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.131@KRBTEST.COM
I20260923 17:23:08.026456 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:08.009558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:38535 (local address 127.23.214.129:59769)
0923 17:23:08.009914 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:08.009928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:08.010040 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:08.012084 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:08.012090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:08.012728 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:08.013572 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:08.013595 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.015571 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.015574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:08.017275 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:08.017287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.018349 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.018356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:08.018551 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:08.019328 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:08.019359 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:08.021503 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:08.023734 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:08.023737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:08.023738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:08.024103 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:08.024517 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:08.024520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:08.024522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:08.024638 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:08.025281 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:08.025290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:08.025543 (+ 253us) client_negotiation.cc:769] Sending connection context
0923 17:23:08.025801 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:23:08.026218 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":398,"spinlock_wait_cycles":29824,"thread_start_us":59,"threads_started":1}
I20260923 17:23:08.026446 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:08.009735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:59769 (local address 127.23.214.131:38535)
0923 17:23:08.010180 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:08.010202 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:08.010246 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:23:08.011143 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:08.011151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:08.011621 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:08.011959 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:08.013790 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.015387 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.017410 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.018179 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.018357 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:08.021772 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:08.021781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:08.021798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:08.021827 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:08.023568 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:08.024246 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:08.024252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:08.024257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:08.024399 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:08.024740 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:08.024743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:08.024745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:08.025155 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:08.025301 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:08.025929 (+ 628us) server_negotiation.cc:299] Negotiation successful
0923 17:23:08.026224 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":127,"threads_started":1}
I20260923 17:23:08.026525 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:08.009753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:42925 (local address 127.23.214.130:33839)
0923 17:23:08.010180 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:08.010197 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:08.010246 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:23:08.011143 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:08.011151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:08.011621 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:08.011959 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:08.013790 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.015387 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.017410 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.018179 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.018357 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:08.021381 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:08.021386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:08.021399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:08.021428 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:08.023530 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:08.024238 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:08.024242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:08.024246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:08.024367 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:08.024728 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:08.024733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:08.024735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:08.025154 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:08.025292 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:08.026000 (+ 708us) server_negotiation.cc:299] Negotiation successful
0923 17:23:08.026328 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":114,"threads_started":1}
I20260923 17:23:08.026511 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:08.009558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33839 (local address 127.23.214.129:42925)
0923 17:23:08.009868 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:08.009914 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:08.010040 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:08.012084 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:08.012090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:08.012728 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:08.013572 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:08.013595 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.015570 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.015574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:08.017275 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:08.017287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:08.018568 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:08.018572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:08.018666 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:08.019407 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:08.019419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:08.021116 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:08.023670 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:08.023678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:08.023694 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:08.024099 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:08.024475 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:08.024477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:08.024479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:08.024629 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:08.025280 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:08.025287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:08.025541 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 17:23:08.025820 (+ 279us) client_negotiation.cc:240] Negotiation successful
0923 17:23:08.026323 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":422,"spinlock_wait_cycles":42624,"thread_start_us":88,"threads_started":1}
W20260923 17:23:08.030447 8569 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
W20260923 17:23:08.030682 8566 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
I20260923 17:23:08.030833 8566 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b37c2506020546d7894d35db6be03f7e; no voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d
I20260923 17:23:08.031153 8970 raft_consensus.cc:2726] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:23:08.330533 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:08.330216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:51453)
0923 17:23:08.330372 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:08.330444 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 17:23:08.479336 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:08.478914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:39943)
0923 17:23:08.479158 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:08.479230 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260923 17:23:08.660717 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:08.660334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:56259)
0923 17:23:08.660567 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:08.660627 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260923 17:23:09.331758 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:09.331296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:60115)
0923 17:23:09.331581 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:09.331654 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260923 17:23:09.480504 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:09.480090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:52857)
0923 17:23:09.480353 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:09.480406 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260923 17:23:09.662124 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:09.661585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:42061)
0923 17:23:09.661964 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:09.662027 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":104,"threads_started":1}
I20260923 17:23:09.966723 8981 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:09.966872 8981 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:09.967338 8981 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
W20260923 17:23:09.968274 8569 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
W20260923 17:23:09.968374 8566 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
I20260923 17:23:09.968448 8566 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b37c2506020546d7894d35db6be03f7e; no voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d
I20260923 17:23:09.968606 8981 raft_consensus.cc:2726] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:23:10.333240 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:10.332775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:35559)
0923 17:23:10.333073 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:10.333141 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260923 17:23:10.481784 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:10.481342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:50987)
0923 17:23:10.481615 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:10.481677 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260923 17:23:10.663550 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:10.663040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:49615)
0923 17:23:10.663373 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:10.663440 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260923 17:23:11.334613 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:11.334167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:44229)
0923 17:23:11.334465 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:11.334519 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":71,"threads_started":1}
I20260923 17:23:11.483078 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:11.482629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:39453)
0923 17:23:11.482910 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:11.482981 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":138,"threads_started":1}
I20260923 17:23:11.664984 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:11.664487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:47783)
0923 17:23:11.664830 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:11.664887 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260923 17:23:11.963613 8988 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:11.963807 8988 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:11.964741 8988 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
W20260923 17:23:11.966017 8569 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
W20260923 17:23:11.966212 8566 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
I20260923 17:23:11.966300 8566 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b37c2506020546d7894d35db6be03f7e; no voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d
I20260923 17:23:11.966504 8988 raft_consensus.cc:2726] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:23:12.335924 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:12.335474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:36719)
0923 17:23:12.335761 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:12.335826 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260923 17:23:12.484385 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:12.483860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:50841)
0923 17:23:12.484210 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:12.484290 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":73,"threads_started":1}
I20260923 17:23:12.666577 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:12.666160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:46053)
0923 17:23:12.666417 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:12.666479 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260923 17:23:13.337229 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:13.336802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:60359)
0923 17:23:13.337066 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:13.337130 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260923 17:23:13.485743 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:13.485284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:44447)
0923 17:23:13.485586 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:13.485640 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260923 17:23:13.667979 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:13.667476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:37497)
0923 17:23:13.667809 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:13.667876 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260923 17:23:14.338511 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:14.338095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:42901)
0923 17:23:14.338355 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:14.338413 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260923 17:23:14.487097 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:14.486560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:51023)
0923 17:23:14.486896 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:14.486970 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
I20260923 17:23:14.669349 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:14.668887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:43437)
0923 17:23:14.669188 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:14.669252 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260923 17:23:15.241444 8998 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:15.241578 8998 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:15.242102 8998 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
W20260923 17:23:15.242940 8566 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
W20260923 17:23:15.243047 8569 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
I20260923 17:23:15.243105 8569 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b37c2506020546d7894d35db6be03f7e; no voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d
I20260923 17:23:15.243264 8998 raft_consensus.cc:2726] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:23:15.339813 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:15.339367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:37925)
0923 17:23:15.339636 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:15.339697 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260923 17:23:15.488379 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:15.487968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:60025)
0923 17:23:15.488225 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:15.488282 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
I20260923 17:23:15.670763 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:15.670297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:35107)
0923 17:23:15.670599 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:15.670666 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260923 17:23:16.341187 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:16.340747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:57889)
0923 17:23:16.341021 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:16.341083 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260923 17:23:16.489723 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:16.489271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:49295)
0923 17:23:16.489526 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:16.489587 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260923 17:23:16.672019 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:16.671596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:37331)
0923 17:23:16.671858 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:16.671921 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260923 17:23:16.956714 9005 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:16.956903 9005 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:16.957418 9005 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
W20260923 17:23:16.958364 8566 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
W20260923 17:23:16.958470 8569 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
I20260923 17:23:16.958536 8569 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b37c2506020546d7894d35db6be03f7e; no voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d
I20260923 17:23:16.958686 9005 raft_consensus.cc:2726] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:23:17.342490 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:17.342071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:42011)
0923 17:23:17.342332 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:17.342393 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260923 17:23:17.490830 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:17.490428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:54525)
0923 17:23:17.490672 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:17.490734 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260923 17:23:17.673305 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:17.672830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:42163)
0923 17:23:17.673149 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:17.673208 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260923 17:23:18.343935 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:18.343461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:52677)
0923 17:23:18.343768 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:18.343834 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260923 17:23:18.492125 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:18.491673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:44319)
0923 17:23:18.491969 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:18.492028 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
I20260923 17:23:18.502939 9011 raft_consensus.cc:482] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:18.503062 9011 raft_consensus.cc:504] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d92582930c4a3f922cf686cb1a4e9d" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 38535 } } peers { permanent_uuid: "b8dddce61c854580bd646a4b4c4287c5" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33839 } } peers { permanent_uuid: "b37c2506020546d7894d35db6be03f7e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 38519 } }
I20260923 17:23:18.503535 9011 leader_election.cc:288] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535), b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839)
W20260923 17:23:18.504338 8569 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1d92582930c4a3f922cf686cb1a4e9d (127.23.214.131:38535): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
W20260923 17:23:18.504454 8566 leader_election.cc:341] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8dddce61c854580bd646a4b4c4287c5 (127.23.214.130:33839): Not found: Tablet not found: 01c8b5659e634f0b9441c9492abd9f19
I20260923 17:23:18.504575 8566 leader_election.cc:302] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b37c2506020546d7894d35db6be03f7e; no voters: b8dddce61c854580bd646a4b4c4287c5, f1d92582930c4a3f922cf686cb1a4e9d
I20260923 17:23:18.504755 9011 raft_consensus.cc:2726] T 01c8b5659e634f0b9441c9492abd9f19 P b37c2506020546d7894d35db6be03f7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:23:18.674782 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:18.674219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:41419)
0923 17:23:18.674523 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:18.674586 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
I20260923 17:23:19.345321 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:19.344849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:47411)
0923 17:23:19.345172 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:19.345226 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":132,"threads_started":1}
I20260923 17:23:19.493407 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:19.492983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:37979)
0923 17:23:19.493246 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:19.493308 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260923 17:23:19.675925 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:19.675519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:42203)
0923 17:23:19.675770 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:19.675825 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260923 17:23:20.346529 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:20.346111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:36201)
0923 17:23:20.346385 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:20.346437 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260923 17:23:20.494594 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:20.494203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:52391)
0923 17:23:20.494440 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:20.494497 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260923 17:23:20.677109 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:20.676698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:36703)
0923 17:23:20.676958 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:20.677016 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260923 17:23:21.347973 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:21.347529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.129:52875)
0923 17:23:21.347814 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:21.347877 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260923 17:23:21.496019 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:21.495623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.130:49433)
0923 17:23:21.495869 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:21.495924 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260923 17:23:21.668241 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 8517
I20260923 17:23:21.674190 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 8684
I20260923 17:23:21.678593 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:21.678185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:34557 (local address 127.23.214.131:40561)
0923 17:23:21.678441 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:21.678497 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.190:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260923 17:23:21.679883 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 8827
2026-09-23T17:23:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59310 ms)
[----------] 30 tests from SecurityITest (248729 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:21 dist-test-slave-9q7f krb5kdc[9024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:21 dist-test-slave-9q7f krb5kdc[9024](info): set up 2 sockets
Sep 23 17:23:21 dist-test-slave-9q7f krb5kdc[9024](info): commencing operation
krb5kdc: starting...
W20260923 17:23:22.743853 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:22 dist-test-slave-9q7f krb5kdc[9024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:22.868616 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:43985
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:22.979158 9037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:22.979451 9037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:22.979496 9037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:22.983503 9037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:22.983570 9037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:22.983589 9037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:22.983611 9037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:22.983628 9037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:22.988834 9037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:22.989996 9037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:22.991045 9037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:23.000151 9043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:23.000204 9045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:23.000211 9037 server_base.cc:1034] running on GCE node
W20260923 17:23:23.000151 9042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:23.000993 9037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:23.002239 9037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:23.003399 9037 hybrid_clock.cc:648] HybridClock initialized: now 1790184203003367 us; error 46 us; skew 500 ppm
Sep 23 17:23:22 dist-test-slave-9q7f krb5kdc[9024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184202, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:23.006134 9037 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:23.007076 9037 webserver.cc:460] Webserver started at http://127.23.214.190:39389/ using document root <none> and password file <none>
I20260923 17:23:23.007663 9037 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:23.007756 9037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:23.008002 9037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:23.010188 9037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "f04182b7233d4137b18f95fc119f5bfa"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "b0c47691617207cb36b1a96aa710e904"
server_key_iv: "07a7510d6fa27748ec6d8ece1e70ea0b"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.010787 9037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "f04182b7233d4137b18f95fc119f5bfa"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "b0c47691617207cb36b1a96aa710e904"
server_key_iv: "07a7510d6fa27748ec6d8ece1e70ea0b"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.015944 9037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:23:23.019760 9052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.021405 9037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:23:23.021620 9037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "f04182b7233d4137b18f95fc119f5bfa"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "b0c47691617207cb36b1a96aa710e904"
server_key_iv: "07a7510d6fa27748ec6d8ece1e70ea0b"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.021785 9037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:23.040859 9037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:23.044097 9037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:23.044315 9037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:23.051829 9037 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:43985
I20260923 17:23:23.051843 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:43985 every 8 connection(s)
I20260923 17:23:23.052843 9037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:23.055032 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9037
I20260923 17:23:23.055138 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:23.055423 24410 external_mini_cluster.cc:1428] Setting key 9aee5cbb4b582de11c9b83408d3ac32e
I20260923 17:23:23.056000 9106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:23.063470 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa: Bootstrap starting.
Sep 23 17:23:23 dist-test-slave-9q7f krb5kdc[9024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:23.066303 9106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:23.067329 9106 log.cc:824] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:23.069767 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa: No bootstrap required, opened a new log
I20260923 17:23:23.071523 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.056859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.23.214.190:43985)
0923 17:23:23.057313 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.057333 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.057375 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:23:23.058280 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.058305 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.058752 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.059138 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:23.059797 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.061326 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.062293 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.062999 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.063165 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.065925 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:23.065932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:23.065950 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:23.065985 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:23.068175 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.068629 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.068635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.068641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.068784 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.069051 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.069054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.069056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.069575 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:23.069783 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.070199 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.070536 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
I20260923 17:23:23.074342 9106 raft_consensus.cc:348] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04182b7233d4137b18f95fc119f5bfa" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43985 } }
I20260923 17:23:23.074579 9106 raft_consensus.cc:374] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:23.074618 9106 raft_consensus.cc:729] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04182b7233d4137b18f95fc119f5bfa, State: Initialized, Role: FOLLOWER
I20260923 17:23:23.075026 9106 consensus_queue.cc:260] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04182b7233d4137b18f95fc119f5bfa" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43985 } }
I20260923 17:23:23.075152 9106 raft_consensus.cc:388] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:23.075192 9106 raft_consensus.cc:482] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:23.075258 9106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:23.076239 9106 raft_consensus.cc:504] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04182b7233d4137b18f95fc119f5bfa" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43985 } }
I20260923 17:23:23.076614 9106 leader_election.cc:302] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f04182b7233d4137b18f95fc119f5bfa; no voters:
I20260923 17:23:23.076996 9106 leader_election.cc:288] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:23.077124 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:23.077435 9111 raft_consensus.cc:686] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [term 1 LEADER]: Becoming Leader. State: Replica: f04182b7233d4137b18f95fc119f5bfa, State: Running, Role: LEADER
I20260923 17:23:23.077776 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04182b7233d4137b18f95fc119f5bfa" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43985 } }
I20260923 17:23:23.078329 9106 sys_catalog.cc:564] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:23.079591 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f04182b7233d4137b18f95fc119f5bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04182b7233d4137b18f95fc119f5bfa" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43985 } } }
I20260923 17:23:23.079602 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f04182b7233d4137b18f95fc119f5bfa. Latest consensus state: current_term: 1 leader_uuid: "f04182b7233d4137b18f95fc119f5bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04182b7233d4137b18f95fc119f5bfa" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 43985 } } }
I20260923 17:23:23.079756 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:23.079756 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:23.080159 9120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:23.083441 9120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:23.089304 9120 catalog_manager.cc:1347] Generated new cluster ID: 08db85c4ebdb4c838c54fa64aa550335
I20260923 17:23:23.089386 9120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:23.095420 9120 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:23.096747 9120 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:23.105930 9120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa: Generated new TSK 0
I20260923 17:23:23.106961 9120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:23:23.145641 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43985
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:23.256754 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:23.257051 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:23.257133 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:23.261183 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:23.261291 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:23.261390 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:23:23.266700 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:23.267799 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:23.268841 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:23.278345 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:23.278345 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:23.278465 9132 server_base.cc:1034] running on GCE node
W20260923 17:23:23.278339 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:23.279083 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:23.279742 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:23.280923 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790184203280897 us; error 42 us; skew 500 ppm
Sep 23 17:23:23 dist-test-slave-9q7f krb5kdc[9024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184203, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:23.283906 9132 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:23:23.284809 9132 webserver.cc:460] Webserver started at http://127.23.214.129:35049/ using document root <none> and password file <none>
I20260923 17:23:23.285379 9132 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:23.285442 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:23.285672 9132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:23.288254 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "434f73e475b14df18c8b417058c5740a"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "ee90ff9f341029f115085c35bd365319"
server_key_iv: "2a2cd2edcd5083391855fe318ddc428d"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.288828 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "434f73e475b14df18c8b417058c5740a"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "ee90ff9f341029f115085c35bd365319"
server_key_iv: "2a2cd2edcd5083391855fe318ddc428d"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.292584 9132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 17:23:23.295011 9147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.296253 9132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:23.296396 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "434f73e475b14df18c8b417058c5740a"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "ee90ff9f341029f115085c35bd365319"
server_key_iv: "2a2cd2edcd5083391855fe318ddc428d"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.296509 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:23.310745 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:23.314075 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:23.314284 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:23.314873 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:23.315697 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:23.315768 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.315851 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:23.315909 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.324295 9132 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:45589
I20260923 17:23:23.324331 9263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:45589 every 8 connection(s)
I20260923 17:23:23.325171 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:23.332468 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9132
I20260923 17:23:23.332635 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:23:23.332959 24410 external_mini_cluster.cc:1428] Setting key c4bad5b51e3a03db3f22761f971c7933
Sep 23 17:23:23 dist-test-slave-9q7f krb5kdc[9024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184203, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:23.339248 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.326966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:52295 (local address 127.23.214.190:43985)
0923 17:23:23.327108 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.327112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.327950 (+ 838us) server_negotiation.cc:407] Connection header received
0923 17:23:23.329336 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.329339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.329398 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.329503 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:23.330685 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.331260 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.332571 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.332892 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.332981 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.335716 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:23.335720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:23.335722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:23.335754 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:23.337179 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.337703 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.337708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.337710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.337789 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.338170 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.338174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.338176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.338365 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:23.338474 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.338956 (+ 482us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.339080 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:23:23.340014 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.327348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43985 (local address 127.23.214.129:52295)
0923 17:23:23.327798 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:23.327841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:23.329094 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:23.329675 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:23.329682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:23.330057 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:23.330537 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.330550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.331409 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.331413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.332411 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.332424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.333055 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.333059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.333252 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.333976 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:23.333992 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:23.335557 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:23.337328 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:23.337334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:23.337349 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:23.337597 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:23.337909 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:23.337915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:23.337920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:23.338063 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:23.338473 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:23.338479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:23.338723 (+ 244us) client_negotiation.cc:769] Sending connection context
0923 17:23:23.338901 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 17:23:23.339218 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
I20260923 17:23:23.341377 9264 heartbeater.cc:347] Connected to a master server at 127.23.214.190:43985
I20260923 17:23:23.341614 9264 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:23:23.342180 9264 heartbeater.cc:511] Master 127.23.214.190:43985 requested a full tablet report, sending...
I20260923 17:23:23.343611 9070 ts_manager.cc:194] Registered new tserver with Master: 434f73e475b14df18c8b417058c5740a (127.23.214.129:45589)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:23:23.344790 9070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:52295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:23:23.359009 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43985
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:23.472167 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:23.472438 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:23.472481 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:23.476649 9269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:23.476713 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:23.476773 9269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:23:23.482421 9269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:23.483479 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:23.484575 9269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:23.494319 9277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:23.494333 9275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:23.494369 9274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:23.494405 9269 server_base.cc:1034] running on GCE node
I20260923 17:23:23.494952 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:23.495563 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:23.496721 9269 hybrid_clock.cc:648] HybridClock initialized: now 1790184203496703 us; error 29 us; skew 500 ppm
Sep 23 17:23:23 dist-test-slave-9q7f krb5kdc[9024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184203, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:23.499413 9269 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:23:23.500234 9269 webserver.cc:460] Webserver started at http://127.23.214.130:34821/ using document root <none> and password file <none>
I20260923 17:23:23.500881 9269 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:23.500942 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:23.501168 9269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:23.503230 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "b35c37b3c4a242caa8d8eb328ea60496"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "b9f2f8d3e43da131e4ecb4562e1d0f44"
server_key_iv: "d2285a53b44afe848c84a7222fd3d74e"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.503779 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "b35c37b3c4a242caa8d8eb328ea60496"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "b9f2f8d3e43da131e4ecb4562e1d0f44"
server_key_iv: "d2285a53b44afe848c84a7222fd3d74e"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.507434 9269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:23:23.509752 9284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.511226 9269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:23.511324 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "b35c37b3c4a242caa8d8eb328ea60496"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "b9f2f8d3e43da131e4ecb4562e1d0f44"
server_key_iv: "d2285a53b44afe848c84a7222fd3d74e"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.511422 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:23.532960 9269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:23.536015 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:23.536186 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:23.536744 9269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:23.537557 9269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:23.537617 9269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.537660 9269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:23.537685 9269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.547263 9269 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:33203
I20260923 17:23:23.547283 9397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:33203 every 8 connection(s)
I20260923 17:23:23.548264 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:23.555774 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9269
I20260923 17:23:23.555917 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:23:23.556288 24410 external_mini_cluster.cc:1428] Setting key 93d8d2f9ce178b1bcec69e7c0437256e
Sep 23 17:23:23 dist-test-slave-9q7f krb5kdc[9024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184203, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:23.562577 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.550157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:37723 (local address 127.23.214.190:43985)
0923 17:23:23.550323 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.550327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.551135 (+ 808us) server_negotiation.cc:407] Connection header received
0923 17:23:23.552418 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.552421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.552467 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.552539 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:23.553632 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.554240 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.555341 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.555550 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.555624 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.558616 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:23.558620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:23.558623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:23.558654 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:23.560401 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.560907 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.560914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.560918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.561014 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.561349 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.561355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.561360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.561622 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:23.561725 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.562282 (+ 557us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.562413 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:23:23.563427 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.550539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43985 (local address 127.23.214.130:37723)
0923 17:23:23.550989 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:23.551031 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:23.552232 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:23.552669 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:23.552676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:23.553034 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:23.553507 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.553521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.554347 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.554350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.555234 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.555243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.555709 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.555714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.555961 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.556637 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:23.556658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:23.558452 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:23.560529 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:23.560534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:23.560548 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:23.560811 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:23.561109 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:23.561111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:23.561113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:23.561243 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:23.561722 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:23.561727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:23.562032 (+ 305us) client_negotiation.cc:769] Sending connection context
0923 17:23:23.562240 (+ 208us) client_negotiation.cc:240] Negotiation successful
0923 17:23:23.562589 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260923 17:23:23.564816 9398 heartbeater.cc:347] Connected to a master server at 127.23.214.190:43985
I20260923 17:23:23.565074 9398 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:23.565569 9398 heartbeater.cc:511] Master 127.23.214.190:43985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:23:23.566773 9070 ts_manager.cc:194] Registered new tserver with Master: b35c37b3c4a242caa8d8eb328ea60496 (127.23.214.130:33203)
I20260923 17:23:23.567365 9070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:37723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:23:23.585239 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43985
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:23.699425 9403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:23.699694 9403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:23.699738 9403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:23.704021 9403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:23.704102 9403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:23.704211 9403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:23:23.709751 9403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:23.710958 9403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:23.712038 9403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:23.721814 9411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:23.721818 9409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:23.722013 9408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:23.722236 9403 server_base.cc:1034] running on GCE node
I20260923 17:23:23.722708 9403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:23.723461 9403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:23.724678 9403 hybrid_clock.cc:648] HybridClock initialized: now 1790184203724644 us; error 50 us; skew 500 ppm
Sep 23 17:23:23 dist-test-slave-9q7f krb5kdc[9024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184203, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:23.728047 9403 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:23:23.728950 9403 webserver.cc:460] Webserver started at http://127.23.214.131:39171/ using document root <none> and password file <none>
I20260923 17:23:23.729540 9403 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:23.729631 9403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:23.729909 9403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:23.732138 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "8a85f3ffb5e046608b558917b09403a1"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "d459f733a28944c6e633c92315dfbb7a"
server_key_iv: "8c0c2c6b671422e4373cd6022522f421"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.732717 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "8a85f3ffb5e046608b558917b09403a1"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "d459f733a28944c6e633c92315dfbb7a"
server_key_iv: "8c0c2c6b671422e4373cd6022522f421"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.736500 9403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:23:23.738911 9418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.740126 9403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:23.740259 9403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "8a85f3ffb5e046608b558917b09403a1"
format_stamp: "Formatted at 2026-09-23 17:23:23 on dist-test-slave-9q7f"
server_key: "d459f733a28944c6e633c92315dfbb7a"
server_key_iv: "8c0c2c6b671422e4373cd6022522f421"
server_key_version: "encryptionkey@0"
I20260923 17:23:23.740381 9403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:23.767338 9403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:23.770943 9403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:23.771232 9403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:23.771867 9403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:23.772818 9403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:23.772892 9403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.772964 9403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:23.773010 9403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:23.782262 9403 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:34525
I20260923 17:23:23.782292 9531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:34525 every 8 connection(s)
I20260923 17:23:23.783246 9403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:23.792038 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9403
I20260923 17:23:23.792204 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:23:23.792534 24410 external_mini_cluster.cc:1428] Setting key fe73dd1988a36eeccc19e3093ff59150
Sep 23 17:23:23 dist-test-slave-9q7f krb5kdc[9024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184203, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:23.797353 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.785139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:40745 (local address 127.23.214.190:43985)
0923 17:23:23.785294 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.785299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.786235 (+ 936us) server_negotiation.cc:407] Connection header received
0923 17:23:23.787597 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.787601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.787657 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.787765 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:23.788923 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.789461 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.790684 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.790910 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.791019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.793710 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:23.793714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:23.793717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:23.793744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:23.795369 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.795906 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.795908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.795909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.795982 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.796273 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.796275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.796284 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.796452 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:23.796527 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.797098 (+ 571us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.797223 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:23:23.798269 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.785538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:43985 (local address 127.23.214.131:40745)
0923 17:23:23.786087 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:23.786131 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:23.787388 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:23.787949 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:23.787956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:23.788300 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:23.788769 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.788791 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.789586 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.789595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.790557 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.790566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.791058 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.791061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.791236 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.791799 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:23.791815 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:23.793541 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:23.795522 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:23.795527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:23.795539 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:23.795818 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:23.796060 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:23.796062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:23.796064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:23.796195 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:23.796536 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:23.796540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:23.796850 (+ 310us) client_negotiation.cc:769] Sending connection context
0923 17:23:23.797033 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 17:23:23.797286 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":122,"threads_started":1}
I20260923 17:23:23.799404 9532 heartbeater.cc:347] Connected to a master server at 127.23.214.190:43985
I20260923 17:23:23.799642 9532 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:23.800136 9532 heartbeater.cc:511] Master 127.23.214.190:43985 requested a full tablet report, sending...
I20260923 17:23:23.801155 9070 ts_manager.cc:194] Registered new tserver with Master: 8a85f3ffb5e046608b558917b09403a1 (127.23.214.131:34525)
I20260923 17:23:23.801714 9070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:40745
I20260923 17:23:23.806558 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:23:23.815248 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.808422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.23.214.190:43985)
0923 17:23:23.808562 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.808567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.808728 (+ 161us) server_negotiation.cc:407] Connection header received
0923 17:23:23.808859 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.808861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.808899 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.808957 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:23.809579 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.810140 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.810941 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.811138 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.811208 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.812510 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:23.812513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:23.812515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:23.812531 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:23.813734 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.814142 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.814145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.814146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.814216 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:23.814419 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:23.814421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:23.814422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:23.814662 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:23.814756 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.814951 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.815037 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:23:23.818145 9070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:23:23.821017 9070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:23:23.835434 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.829333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45589 (local address 127.0.0.1:50246)
0923 17:23:23.830817 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:23.830828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:23.830937 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:23.831427 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:23.831430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:23.831445 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:23.831729 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.831772 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.832872 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.832875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.834068 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.834078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.834872 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.834876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.834982 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.835017 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:23:23.835128 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:23:23.835247 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"mutex_wait_us":21,"thread_start_us":109,"threads_started":1}
I20260923 17:23:23.835575 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.829537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.23.214.131:34525)
0923 17:23:23.829951 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.829965 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.830447 (+ 482us) server_negotiation.cc:407] Connection header received
0923 17:23:23.830530 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.830535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.830745 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.830907 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:23.831218 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.832355 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.834145 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.834992 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.835111 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.835225 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.835319 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.835416 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":137,"threads_started":1}
I20260923 17:23:23.836038 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.830284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.23.214.130:33203)
0923 17:23:23.830793 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.830809 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.831024 (+ 215us) server_negotiation.cc:407] Connection header received
0923 17:23:23.831237 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.831241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.831410 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.831551 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:23.831976 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.832983 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.834219 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.834992 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.835111 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.835224 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.835803 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.835882 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":159,"threads_started":1}
I20260923 17:23:23.836050 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.830026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.23.214.129:45589)
0923 17:23:23.830678 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:23.830694 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:23.831025 (+ 331us) server_negotiation.cc:407] Connection header received
0923 17:23:23.831102 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:23.831108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:23.831291 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:23.831435 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:23.831896 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.832722 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.834181 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.834732 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.834888 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.835029 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:23.835834 (+ 805us) server_negotiation.cc:299] Negotiation successful
0923 17:23:23.835922 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":92,"threads_started":1}
I20260923 17:23:23.836660 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.830316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33203 (local address 127.0.0.1:51438)
0923 17:23:23.830942 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:23.830956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:23.831099 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:23.831713 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:23.831716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:23.831733 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:23.831843 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.831849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.833124 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.833127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.834066 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.834076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.835292 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.835296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.835392 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.835414 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:23:23.835484 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 17:23:23.836257 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260923 17:23:23.835436 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:23.829265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:34525 (local address 127.0.0.1:53386)
0923 17:23:23.830086 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:23.830138 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:23.830270 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:23.830924 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:23.830930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:23.830949 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:23.831097 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.831104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.832490 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.832493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.834022 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:23.834033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:23.835107 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:23.835113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:23.835196 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:23.835216 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:23:23.835288 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:23:23.835347 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":32,"thread_start_us":81,"threads_started":1}
I20260923 17:23:23.839295 9466 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6b6a58ac0446b984e5a0bb1df874b3 (DEFAULT_TABLE table=test-table [id=7e07b7cd54424c8c9fe5450c19b51952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:23.839308 9195 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6b6a58ac0446b984e5a0bb1df874b3 (DEFAULT_TABLE table=test-table [id=7e07b7cd54424c8c9fe5450c19b51952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:23.839290 9332 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6b6a58ac0446b984e5a0bb1df874b3 (DEFAULT_TABLE table=test-table [id=7e07b7cd54424c8c9fe5450c19b51952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:23.840360 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b6b6a58ac0446b984e5a0bb1df874b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:23.840363 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b6b6a58ac0446b984e5a0bb1df874b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:23.840380 9332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b6b6a58ac0446b984e5a0bb1df874b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:23.847096 9549 tablet_bootstrap.cc:492] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Bootstrap starting.
I20260923 17:23:23.848245 9551 tablet_bootstrap.cc:492] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Bootstrap starting.
I20260923 17:23:23.849716 9549 tablet_bootstrap.cc:654] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:23.850045 9551 tablet_bootstrap.cc:654] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:23.850212 9550 tablet_bootstrap.cc:492] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a: Bootstrap starting.
I20260923 17:23:23.851351 9551 log.cc:824] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:23.851429 9549 log.cc:824] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:23.852360 9550 tablet_bootstrap.cc:654] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:23.852996 9550 log.cc:824] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:23.853461 9549 tablet_bootstrap.cc:492] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: No bootstrap required, opened a new log
I20260923 17:23:23.853457 9551 tablet_bootstrap.cc:492] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: No bootstrap required, opened a new log
I20260923 17:23:23.853658 9549 ts_tablet_manager.cc:1397] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 17:23:23.853658 9551 ts_tablet_manager.cc:1397] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 17:23:23.854755 9550 tablet_bootstrap.cc:492] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a: No bootstrap required, opened a new log
I20260923 17:23:23.854959 9550 ts_tablet_manager.cc:1397] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:23:23.857673 9549 raft_consensus.cc:348] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:23.857673 9551 raft_consensus.cc:348] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:23.857980 9551 raft_consensus.cc:374] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:23.857980 9549 raft_consensus.cc:374] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:23.858044 9549 raft_consensus.cc:729] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a85f3ffb5e046608b558917b09403a1, State: Initialized, Role: FOLLOWER
I20260923 17:23:23.858373 9551 raft_consensus.cc:729] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c37b3c4a242caa8d8eb328ea60496, State: Initialized, Role: FOLLOWER
I20260923 17:23:23.858372 9550 raft_consensus.cc:348] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:23.858534 9549 consensus_queue.cc:260] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:23.858656 9550 raft_consensus.cc:374] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:23.858714 9550 raft_consensus.cc:729] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434f73e475b14df18c8b417058c5740a, State: Initialized, Role: FOLLOWER
I20260923 17:23:23.858877 9551 consensus_queue.cc:260] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:23.859148 9550 consensus_queue.cc:260] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:23.859302 9532 heartbeater.cc:503] Master 127.23.214.190:43985 was elected leader, sending a full tablet report...
I20260923 17:23:23.859642 9549 ts_tablet_manager.cc:1428] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:23:23.859722 9551 ts_tablet_manager.cc:1428] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260923 17:23:23.860051 9398 heartbeater.cc:503] Master 127.23.214.190:43985 was elected leader, sending a full tablet report...
I20260923 17:23:23.860070 9550 ts_tablet_manager.cc:1428] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:23:23.860522 9264 heartbeater.cc:503] Master 127.23.214.190:43985 was elected leader, sending a full tablet report...
W20260923 17:23:24.034297 9533 tablet.cc:2369] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:23:24.049521 9399 tablet.cc:2369] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:23:24.076505 9265 tablet.cc:2369] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:24.140630 9555 raft_consensus.cc:482] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:24.140915 9555 raft_consensus.cc:504] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:24.142071 9555 leader_election.cc:288] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 434f73e475b14df18c8b417058c5740a (127.23.214.129:45589), b35c37b3c4a242caa8d8eb328ea60496 (127.23.214.130:33203)
I20260923 17:23:24.146042 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.142523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:41777 (local address 127.23.214.129:45589)
0923 17:23:24.142675 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:24.142680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:24.142723 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:23:24.142795 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:24.142800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:24.142859 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:24.142940 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:24.143227 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.143799 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.145011 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.145452 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.145586 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.145644 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:24.145821 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 17:23:24.145918 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:23:24.146042 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.142383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45589 (local address 127.23.214.131:41777)
0923 17:23:24.142522 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:24.142539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:24.142655 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:24.142971 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:24.142974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:24.142993 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:24.143114 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.143120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.143910 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.143916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.144864 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.144873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.145575 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.145579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.145691 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.145710 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 17:23:24.145806 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:23:24.145905 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:23:24.146293 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.142793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:49567 (local address 127.23.214.130:33203)
0923 17:23:24.142939 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:24.142943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:24.143105 (+ 162us) server_negotiation.cc:407] Connection header received
0923 17:23:24.143253 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:24.143256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:24.143306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:24.143395 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:24.143689 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.144224 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.145318 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.145700 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.145835 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.145903 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:24.146074 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 17:23:24.146158 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:23:24.146473 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.142642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:33203 (local address 127.23.214.131:49567)
0923 17:23:24.143013 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:24.143024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:24.143138 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:24.143420 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:24.143422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:24.143438 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:24.143595 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.143601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.144331 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.144335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.145197 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.145206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.145818 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.145822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.145942 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.145969 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:23:24.146057 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:23:24.146256 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":161,"threads_started":1}
I20260923 17:23:24.146749 9216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6b6a58ac0446b984e5a0bb1df874b3" candidate_uuid: "8a85f3ffb5e046608b558917b09403a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434f73e475b14df18c8b417058c5740a" is_pre_election: true
I20260923 17:23:24.147101 9216 raft_consensus.cc:2454] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a85f3ffb5e046608b558917b09403a1 in term 0.
I20260923 17:23:24.147073 9352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6b6a58ac0446b984e5a0bb1df874b3" candidate_uuid: "8a85f3ffb5e046608b558917b09403a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c37b3c4a242caa8d8eb328ea60496" is_pre_election: true
I20260923 17:23:24.147332 9352 raft_consensus.cc:2454] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a85f3ffb5e046608b558917b09403a1 in term 0.
I20260923 17:23:24.147655 9420 leader_election.cc:302] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 434f73e475b14df18c8b417058c5740a, 8a85f3ffb5e046608b558917b09403a1; no voters:
I20260923 17:23:24.147967 9555 raft_consensus.cc:2781] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:23:24.148041 9555 raft_consensus.cc:482] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:23:24.148072 9555 raft_consensus.cc:3037] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:24.149123 9555 raft_consensus.cc:504] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:24.149468 9555 leader_election.cc:288] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [CANDIDATE]: Term 1 election: Requested vote from peers 434f73e475b14df18c8b417058c5740a (127.23.214.129:45589), b35c37b3c4a242caa8d8eb328ea60496 (127.23.214.130:33203)
I20260923 17:23:24.149844 9216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6b6a58ac0446b984e5a0bb1df874b3" candidate_uuid: "8a85f3ffb5e046608b558917b09403a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434f73e475b14df18c8b417058c5740a"
I20260923 17:23:24.149951 9352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6b6a58ac0446b984e5a0bb1df874b3" candidate_uuid: "8a85f3ffb5e046608b558917b09403a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c37b3c4a242caa8d8eb328ea60496"
I20260923 17:23:24.150031 9216 raft_consensus.cc:3037] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:24.150089 9352 raft_consensus.cc:3037] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:24.151117 9352 raft_consensus.cc:2454] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a85f3ffb5e046608b558917b09403a1 in term 1.
I20260923 17:23:24.151124 9216 raft_consensus.cc:2454] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a85f3ffb5e046608b558917b09403a1 in term 1.
I20260923 17:23:24.151453 9420 leader_election.cc:302] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 434f73e475b14df18c8b417058c5740a, 8a85f3ffb5e046608b558917b09403a1; no voters:
I20260923 17:23:24.151633 9555 raft_consensus.cc:2781] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:24.151855 9555 raft_consensus.cc:686] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8a85f3ffb5e046608b558917b09403a1, State: Running, Role: LEADER
I20260923 17:23:24.152184 9555 consensus_queue.cc:237] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } }
I20260923 17:23:24.155273 9068 catalog_manager.cc:5495] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a85f3ffb5e046608b558917b09403a1 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "8a85f3ffb5e046608b558917b09403a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a85f3ffb5e046608b558917b09403a1" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 } health_report { overall_health: UNKNOWN } } }
I20260923 17:23:24.198705 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.195402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.23.214.131:34525)
0923 17:23:24.195534 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:24.195537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:24.195577 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:23:24.195653 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:24.195655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:24.195713 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:24.195801 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:23:24.196005 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.196503 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.197310 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.197525 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.197598 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.197902 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:23:24.198373 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:23:24.198466 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:24.198525 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 17:23:24.198573 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:23:24.204843 9216 raft_consensus.cc:1261] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 434f73e475b14df18c8b417058c5740a [term 1 FOLLOWER]: Refusing update from remote peer 8a85f3ffb5e046608b558917b09403a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:24.204859 9352 raft_consensus.cc:1261] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 1 FOLLOWER]: Refusing update from remote peer 8a85f3ffb5e046608b558917b09403a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:24.205564 9559 consensus_queue.cc:1035] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "434f73e475b14df18c8b417058c5740a" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 45589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:24.205804 9555 consensus_queue.cc:1035] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c37b3c4a242caa8d8eb328ea60496" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:24.213488 9562 mvcc.cc:205] Tried to move back new op lower bound from 7332594500415021056 to 7332594500209762304. Current Snapshot: MvccSnapshot[applied={T|T < 7332594500415021056}]
I20260923 17:23:24.214483 9564 mvcc.cc:205] Tried to move back new op lower bound from 7332594500415021056 to 7332594500209762304. Current Snapshot: MvccSnapshot[applied={T|T < 7332594500415021056}]
I20260923 17:23:24.222539 9069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:24.222729 9069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:24.225183 9069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f04182b7233d4137b18f95fc119f5bfa: Sending DeleteTablet for 3 replicas of tablet 7b6b6a58ac0446b984e5a0bb1df874b3
I20260923 17:23:24.226011 9466 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6b6a58ac0446b984e5a0bb1df874b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:53386
I20260923 17:23:24.226079 9195 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6b6a58ac0446b984e5a0bb1df874b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:50246
I20260923 17:23:24.226063 9332 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6b6a58ac0446b984e5a0bb1df874b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:24 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:51438
I20260923 17:23:24.226608 9574 tablet_replica.cc:330] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: stopping tablet replica
I20260923 17:23:24.226965 9574 raft_consensus.cc:2229] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:24.227257 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9132
I20260923 17:23:24.229585 9574 raft_consensus.cc:2258] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:24.231380 9574 ts_tablet_manager.cc:1905] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:24.234117 9573 tablet_replica.cc:330] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: stopping tablet replica
I20260923 17:23:24.234515 9573 raft_consensus.cc:2229] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:23:24.235025 9574 ts_tablet_manager.cc:1918] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:24.235046 9573 raft_consensus.cc:2258] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:24.235131 9574 log.cc:1196] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/7b6b6a58ac0446b984e5a0bb1df874b3
W20260923 17:23:24.235106 9054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:45589 (error 108)
I20260923 17:23:24.235469 9574 ts_tablet_manager.cc:1939] T 7b6b6a58ac0446b984e5a0bb1df874b3 P b35c37b3c4a242caa8d8eb328ea60496: Deleting consensus metadata
I20260923 17:23:24.236598 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.236258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:45589 (local address 127.0.0.1:50258)
0923 17:23:24.236433 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:24.236497 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260923 17:23:24.236796 9054 catalog_manager.cc:4584] TS 434f73e475b14df18c8b417058c5740a (127.23.214.129:45589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b6b6a58ac0446b984e5a0bb1df874b3: Network error: Client connection negotiation failed: client connection to 127.23.214.129:45589: connect: Connection refused (error 111)
I20260923 17:23:24.236943 9573 ts_tablet_manager.cc:1905] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:24.237067 9054 catalog_manager.cc:4841] TS b35c37b3c4a242caa8d8eb328ea60496 (127.23.214.130:33203): tablet 7b6b6a58ac0446b984e5a0bb1df874b3 (table test-table [id=7e07b7cd54424c8c9fe5450c19b51952]) successfully deleted
I20260923 17:23:24.237407 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9269
I20260923 17:23:24.239969 9573 ts_tablet_manager.cc:1918] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:24.240070 9573 log.cc:1196] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/7b6b6a58ac0446b984e5a0bb1df874b3
I20260923 17:23:24.240386 9573 ts_tablet_manager.cc:1939] T 7b6b6a58ac0446b984e5a0bb1df874b3 P 8a85f3ffb5e046608b558917b09403a1: Deleting consensus metadata
I20260923 17:23:24.241701 9056 catalog_manager.cc:4841] TS 8a85f3ffb5e046608b558917b09403a1 (127.23.214.131:34525): tablet 7b6b6a58ac0446b984e5a0bb1df874b3 (table test-table [id=7e07b7cd54424c8c9fe5450c19b51952]) successfully deleted
I20260923 17:23:24.244635 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9403
I20260923 17:23:24.252231 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9037
2026-09-23T17:23:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2573 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T17:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:24Z Disabled control of system clock
I20260923 17:23:24.283308 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44497
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:23:24.393301 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:24.393570 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:24.393606 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:24.397657 9579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:24.397734 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:24.397749 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:24.397768 9579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:24.397789 9579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:24.402683 9579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:44497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:24.403700 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:24.405033 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:24.413324 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:24.413339 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:24.413349 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:24.413600 9579 server_base.cc:1034] running on GCE node
I20260923 17:23:24.414268 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:24.415428 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:24.416625 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790184204416604 us; error 35 us; skew 500 ppm
I20260923 17:23:24.418567 9579 webserver.cc:460] Webserver started at http://127.23.214.190:35699/ using document root <none> and password file <none>
I20260923 17:23:24.419147 9579 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:24.419239 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:24.419478 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:24.421552 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "4aa8a0265c44400eae09a9cc31b9b9e5"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "859ea885a8a619be0faa07e8420455ab"
server_key_iv: "88bfa9a7a20f2da4ab23798f46fd3031"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.422132 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "4aa8a0265c44400eae09a9cc31b9b9e5"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "859ea885a8a619be0faa07e8420455ab"
server_key_iv: "88bfa9a7a20f2da4ab23798f46fd3031"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.426012 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:23:24.428517 9593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:24.429762 9579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:24.429942 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "4aa8a0265c44400eae09a9cc31b9b9e5"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "859ea885a8a619be0faa07e8420455ab"
server_key_iv: "88bfa9a7a20f2da4ab23798f46fd3031"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.430066 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:24.451368 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:24.452106 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:24.452316 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:24.459239 9579 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:44497
I20260923 17:23:24.459249 9645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:44497 every 8 connection(s)
I20260923 17:23:24.460477 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:24.463689 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:24.469475 9646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5: Bootstrap starting.
I20260923 17:23:24.469609 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9579
I20260923 17:23:24.469759 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:24.470095 24410 external_mini_cluster.cc:1428] Setting key afb482af828c339425802dc2682e7f81
I20260923 17:23:24.472361 9646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:24.473276 9646 log.cc:824] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:24.475507 9646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5: No bootstrap required, opened a new log
I20260923 17:23:24.480036 9646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44497 } }
I20260923 17:23:24.480314 9646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:24.480370 9646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa8a0265c44400eae09a9cc31b9b9e5, State: Initialized, Role: FOLLOWER
I20260923 17:23:24.480892 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.471557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.23.214.190:44497)
0923 17:23:24.472276 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:24.472312 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:24.472358 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:23:24.473240 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:24.473265 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:24.473791 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:24.474227 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:24.474662 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.476085 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.476990 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.477693 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.477843 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.478712 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:24.478718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:24.478737 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:24.479342 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:24.479423 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:24.479450 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:24.479550 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:24.479690 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 17:23:24.479954 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260923 17:23:24.480837 9646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44497 } }
I20260923 17:23:24.480988 9646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:24.481041 9646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:24.481143 9646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:24.482239 9646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44497 } }
I20260923 17:23:24.482582 9646 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aa8a0265c44400eae09a9cc31b9b9e5; no voters:
I20260923 17:23:24.482908 9646 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:24.482972 9651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:24.483196 9651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [term 1 LEADER]: Becoming Leader. State: Replica: 4aa8a0265c44400eae09a9cc31b9b9e5, State: Running, Role: LEADER
I20260923 17:23:24.483484 9651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44497 } }
I20260923 17:23:24.484170 9646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:24.484645 9651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa8a0265c44400eae09a9cc31b9b9e5. Latest consensus state: current_term: 1 leader_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44497 } } }
I20260923 17:23:24.484757 9651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:24.484972 9651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8a0265c44400eae09a9cc31b9b9e5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44497 } } }
I20260923 17:23:24.485072 9651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:24.485249 9655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:24.488056 9655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:24.493314 9655 catalog_manager.cc:1347] Generated new cluster ID: 3df84be39fe547d6ad519b3c6828f3c9
I20260923 17:23:24.493376 9655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:24.520614 9655 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:24.521909 9655 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:24.529610 9655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5: Generated new TSK 0
I20260923 17:23:24.530471 9655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:24.541349 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44497
--builtin_ntp_servers=127.23.214.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:23:24.649197 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:24.649464 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:24.649534 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:24.653555 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:24.653637 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:24.653729 9670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:23:24.658382 9670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:24.659418 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:24.660765 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:24.670217 9676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:24.670239 9675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:24.670418 9670 server_base.cc:1034] running on GCE node
W20260923 17:23:24.670240 9678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:24.670945 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:24.671551 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:24.672737 9670 hybrid_clock.cc:648] HybridClock initialized: now 1790184204672705 us; error 43 us; skew 500 ppm
I20260923 17:23:24.674571 9670 webserver.cc:460] Webserver started at http://127.23.214.129:40125/ using document root <none> and password file <none>
I20260923 17:23:24.675135 9670 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:24.675230 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:24.675457 9670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:24.677449 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "d612684f67374fc1ad53215f121ee90c"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "ab81cae9aec99e21c675d91dbfd842f1"
server_key_iv: "6759a09fd7342f89c93fa69b7aa166bf"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.678071 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "d612684f67374fc1ad53215f121ee90c"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "ab81cae9aec99e21c675d91dbfd842f1"
server_key_iv: "6759a09fd7342f89c93fa69b7aa166bf"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.681507 9670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:23:24.683826 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:24.685158 9670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:24.685286 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "d612684f67374fc1ad53215f121ee90c"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "ab81cae9aec99e21c675d91dbfd842f1"
server_key_iv: "6759a09fd7342f89c93fa69b7aa166bf"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.685393 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:24.694725 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:24.695266 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:24.695398 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:24.695871 9670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:24.696676 9670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:24.696720 9670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:24.696758 9670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:24.696784 9670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:24.704977 9670 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:37289
I20260923 17:23:24.705058 9798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:37289 every 8 connection(s)
I20260923 17:23:24.705909 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:24.707430 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9670
I20260923 17:23:24.707559 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:23:24.707875 24410 external_mini_cluster.cc:1428] Setting key 81abe0c384e3b40bec5ff33795f268db
I20260923 17:23:24.709867 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44497
--builtin_ntp_servers=127.23.214.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:23:24.716257 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.709200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:47541 (local address 127.23.214.190:44497)
0923 17:23:24.709323 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:24.709326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:24.709338 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:23:24.710873 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:24.710877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:24.710928 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:24.711015 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:24.711893 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.712757 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.714015 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.714455 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.714588 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.715654 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:24.715660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:24.715665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:24.715683 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:24.715711 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:24.715725 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:24.715823 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:24.715974 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 17:23:24.716051 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:23:24.716979 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.708274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44497 (local address 127.23.214.129:47541)
0923 17:23:24.709052 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:24.709105 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:24.710661 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:24.711197 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:24.711204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:24.711283 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:24.711752 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.711765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.712891 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.712894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.713841 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.713850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.714601 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.714604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.714747 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.715390 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:24.715391 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:24.715430 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:24.715491 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:24.715828 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:24.715834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:24.715839 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:23:24.715933 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:23:24.716148 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":133,"threads_started":1}
I20260923 17:23:24.718302 9799 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44497
I20260923 17:23:24.718565 9799 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:24.719237 9799 heartbeater.cc:511] Master 127.23.214.190:44497 requested a full tablet report, sending...
I20260923 17:23:24.720942 9610 ts_manager.cc:194] Registered new tserver with Master: d612684f67374fc1ad53215f121ee90c (127.23.214.129:37289)
I20260923 17:23:24.722179 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.129:47541
W20260923 17:23:24.821211 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:24.821460 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:24.821498 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:24.825583 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:24.825636 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:24.825690 9802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:23:24.830435 9802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:24.831454 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:24.832813 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:24.842404 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:24.842528 9802 server_base.cc:1034] running on GCE node
W20260923 17:23:24.842404 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:24.842465 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:24.842983 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:24.843592 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:24.844775 9802 hybrid_clock.cc:648] HybridClock initialized: now 1790184204844759 us; error 26 us; skew 500 ppm
I20260923 17:23:24.846895 9802 webserver.cc:460] Webserver started at http://127.23.214.130:43481/ using document root <none> and password file <none>
I20260923 17:23:24.847472 9802 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:24.847532 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:24.847783 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:24.849841 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "965a5aa577f248719e0579079028b51e"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "b824866519ab84a6113d668fb260df59"
server_key_iv: "b29e6ea754a9acd2b2e51ee4ed8a11af"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.850512 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "965a5aa577f248719e0579079028b51e"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "b824866519ab84a6113d668fb260df59"
server_key_iv: "b29e6ea754a9acd2b2e51ee4ed8a11af"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.854068 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:24.856509 9816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:24.857703 9802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:24.857832 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "965a5aa577f248719e0579079028b51e"
format_stamp: "Formatted at 2026-09-23 17:23:24 on dist-test-slave-9q7f"
server_key: "b824866519ab84a6113d668fb260df59"
server_key_iv: "b29e6ea754a9acd2b2e51ee4ed8a11af"
server_key_version: "encryptionkey@0"
I20260923 17:23:24.857990 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:24.874505 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:24.875209 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:24.875415 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:24.876026 9802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:24.876972 9802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:24.877041 9802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:24.877113 9802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:24.877156 9802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:24.886363 9802 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:45609
I20260923 17:23:24.886389 9930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:45609 every 8 connection(s)
I20260923 17:23:24.887228 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:24.895875 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.889106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:48431 (local address 127.23.214.190:44497)
0923 17:23:24.889239 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:24.889242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:24.890275 (+ 1033us) server_negotiation.cc:407] Connection header received
0923 17:23:24.891582 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:24.891585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:24.891630 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:24.891702 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:24.892528 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.893040 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.894113 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.894321 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.894397 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.895491 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:24.895493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:24.895495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:24.895506 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:24.895520 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:24.895527 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:24.895578 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:24.895700 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 17:23:24.895751 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:23:24.896492 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9802
I20260923 17:23:24.896628 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:23:24.896806 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:24.889489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44497 (local address 127.23.214.130:48431)
0923 17:23:24.890126 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:24.890167 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:24.891402 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:24.891856 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:24.891863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:24.891943 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:24.892409 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.892421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.893139 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.893141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.894016 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:24.894024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:24.894442 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:24.894444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:24.894578 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:24.895263 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:24.895264 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:24.895306 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:24.895367 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:24.895588 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:24.895592 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:24.895596 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:23:24.895680 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:23:24.895902 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
I20260923 17:23:24.896971 24410 external_mini_cluster.cc:1428] Setting key 920eac4f3381ae8c3b174ca5984af573
I20260923 17:23:24.898077 9931 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44497
I20260923 17:23:24.898337 9931 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:24.898851 9931 heartbeater.cc:511] Master 127.23.214.190:44497 requested a full tablet report, sending...
I20260923 17:23:24.899029 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44497
--builtin_ntp_servers=127.23.214.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:23:24.899946 9610 ts_manager.cc:194] Registered new tserver with Master: 965a5aa577f248719e0579079028b51e (127.23.214.130:45609)
I20260923 17:23:24.900758 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.130:48431
W20260923 17:23:25.010439 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:25.010694 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:25.010761 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:25.014772 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:25.014822 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:25.014932 9934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:23:25.019558 9934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:25.020519 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:25.021775 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:25.031602 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:25.031586 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:25.032104 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:25.032248 9934 server_base.cc:1034] running on GCE node
I20260923 17:23:25.032810 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:25.033633 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:25.034849 9934 hybrid_clock.cc:648] HybridClock initialized: now 1790184205034794 us; error 85 us; skew 500 ppm
I20260923 17:23:25.045029 9934 webserver.cc:460] Webserver started at http://127.23.214.131:32775/ using document root <none> and password file <none>
I20260923 17:23:25.045665 9934 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:25.045732 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:25.045964 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:25.048004 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "d145ce7c987e4e88a480d091bf7fc2df"
format_stamp: "Formatted at 2026-09-23 17:23:25 on dist-test-slave-9q7f"
server_key: "d5598335af874542ca357c74da87657d"
server_key_iv: "1260d365b83f5d47addf08075a4fa211"
server_key_version: "encryptionkey@0"
I20260923 17:23:25.048518 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "d145ce7c987e4e88a480d091bf7fc2df"
format_stamp: "Formatted at 2026-09-23 17:23:25 on dist-test-slave-9q7f"
server_key: "d5598335af874542ca357c74da87657d"
server_key_iv: "1260d365b83f5d47addf08075a4fa211"
server_key_version: "encryptionkey@0"
I20260923 17:23:25.052129 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 17:23:25.054459 9948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:25.055680 9934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:25.055765 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "d145ce7c987e4e88a480d091bf7fc2df"
format_stamp: "Formatted at 2026-09-23 17:23:25 on dist-test-slave-9q7f"
server_key: "d5598335af874542ca357c74da87657d"
server_key_iv: "1260d365b83f5d47addf08075a4fa211"
server_key_version: "encryptionkey@0"
I20260923 17:23:25.055843 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:25.070618 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:25.071202 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:25.071354 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:25.071837 9934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:25.072671 9934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:25.072721 9934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:25.072759 9934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:25.072781 9934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:25.081461 9934 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:37025
I20260923 17:23:25.081485 10062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:37025 every 8 connection(s)
I20260923 17:23:25.082403 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:25.086128 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 9934
I20260923 17:23:25.086267 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:23:25.086550 24410 external_mini_cluster.cc:1428] Setting key ff73a91f85ad6f68e01f565ef0ad4f57
I20260923 17:23:25.092181 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.084181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:54237 (local address 127.23.214.190:44497)
0923 17:23:25.084331 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.084336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.085481 (+ 1145us) server_negotiation.cc:407] Connection header received
0923 17:23:25.087157 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.087161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.087217 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.087300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:25.088633 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.089218 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.090524 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.090737 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.090823 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.091776 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:25.091780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:25.091782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:25.091793 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:25.091820 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:25.091828 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:25.091883 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.092019 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.092068 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:23:25.093047 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.084613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44497 (local address 127.23.214.131:54237)
0923 17:23:25.085343 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:25.085384 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:25.086936 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:25.087531 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:25.087540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:25.087656 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:25.088350 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.088369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.089336 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.089338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.090423 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.090433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.090869 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.090871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.091006 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.091546 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:25.091548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:25.091587 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:25.091649 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:25.091900 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:25.091906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:25.091910 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:23:25.092003 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 17:23:25.092247 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260923 17:23:25.094120 10063 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44497
I20260923 17:23:25.094344 10063 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:25.094836 10063 heartbeater.cc:511] Master 127.23.214.190:44497 requested a full tablet report, sending...
I20260923 17:23:25.095806 9610 ts_manager.cc:194] Registered new tserver with Master: d145ce7c987e4e88a480d091bf7fc2df (127.23.214.131:37025)
I20260923 17:23:25.096326 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.131:54237
I20260923 17:23:25.100404 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:23:25.106822 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.102568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.23.214.190:44497)
0923 17:23:25.102732 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.102738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.102923 (+ 185us) server_negotiation.cc:407] Connection header received
0923 17:23:25.103103 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.103107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.103164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.103255 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:25.103551 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.104100 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.104983 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.105231 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.105347 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.106301 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:25.106306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:25.106309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:25.106324 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:25.106354 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:25.106364 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:25.106451 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.106578 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.106650 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:23:25.109663 9610 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:23:25.112638 9610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:23:25.126904 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.120763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:45609 (local address 127.0.0.1:56448)
0923 17:23:25.121820 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:25.121836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:25.121952 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:25.122851 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:25.122854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:25.122871 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:25.122955 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.122965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.123939 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.123942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.125454 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.125465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.126322 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.126325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.126465 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.126508 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:23:25.126607 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:23:25.126723 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":84,"thread_start_us":120,"threads_started":1}
I20260923 17:23:25.126955 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.121455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.23.214.130:45609)
0923 17:23:25.122462 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.122473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.122493 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:23:25.122559 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.122564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.122721 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.122862 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:25.123084 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.123830 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.125621 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.126181 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.126477 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.126638 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.126683 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.126798 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":98,"threads_started":1}
I20260923 17:23:25.127079 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.121797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.23.214.131:37025)
0923 17:23:25.122181 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.122194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.122217 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:23:25.122342 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.122347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.122525 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.122665 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:25.122919 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.124001 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.125577 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.126339 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.126459 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.126593 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.126825 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.126919 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":181,"threads_started":1}
I20260923 17:23:25.127526 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.120690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37289 (local address 127.0.0.1:56650)
0923 17:23:25.121279 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:25.121328 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:25.121454 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:25.122268 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:25.122272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:25.122292 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:25.122386 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.122391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.123842 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.123845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.125226 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.125235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.126299 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.126305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.127213 (+ 908us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.127236 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:23:25.127313 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:23:25.127391 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":304,"thread_start_us":61,"threads_started":1}
I20260923 17:23:25.127621 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.120816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.23.214.129:37289)
0923 17:23:25.121147 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.121159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.121602 (+ 443us) server_negotiation.cc:407] Connection header received
0923 17:23:25.121690 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.121695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.121951 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.122107 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:25.122507 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.123720 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.125354 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.126159 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.126289 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.126491 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.127351 (+ 860us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.127460 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
I20260923 17:23:25.127074 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.121403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37025 (local address 127.0.0.1:47674)
0923 17:23:25.122136 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:25.122148 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:25.122223 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:25.122666 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:25.122669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:25.122687 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:25.122779 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.122795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.124118 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.124124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.125454 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.125465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.126564 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.126568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.126658 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.126687 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:23:25.126796 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:23:25.126876 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":80,"thread_start_us":94,"threads_started":1}
I20260923 17:23:25.130179 9732 tablet_service.cc:1467] Processing CreateTablet for tablet e5e3aacbc68e47f290c75a69d378bc06 (DEFAULT_TABLE table=test-table [id=ba131cd5a20d43818874e9cb57b237a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:25.130228 9865 tablet_service.cc:1467] Processing CreateTablet for tablet e5e3aacbc68e47f290c75a69d378bc06 (DEFAULT_TABLE table=test-table [id=ba131cd5a20d43818874e9cb57b237a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:25.130228 9997 tablet_service.cc:1467] Processing CreateTablet for tablet e5e3aacbc68e47f290c75a69d378bc06 (DEFAULT_TABLE table=test-table [id=ba131cd5a20d43818874e9cb57b237a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:25.131163 9732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e3aacbc68e47f290c75a69d378bc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:25.131269 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e3aacbc68e47f290c75a69d378bc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:25.131287 9865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e3aacbc68e47f290c75a69d378bc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:25.137809 10081 tablet_bootstrap.cc:492] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c: Bootstrap starting.
I20260923 17:23:25.139031 10082 tablet_bootstrap.cc:492] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Bootstrap starting.
I20260923 17:23:25.139460 10083 tablet_bootstrap.cc:492] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Bootstrap starting.
I20260923 17:23:25.139948 10081 tablet_bootstrap.cc:654] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:25.140651 10081 log.cc:824] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:25.141059 10082 tablet_bootstrap.cc:654] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:25.141420 10083 tablet_bootstrap.cc:654] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:25.141916 10082 log.cc:824] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:25.142189 10083 log.cc:824] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:25.143366 10081 tablet_bootstrap.cc:492] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c: No bootstrap required, opened a new log
I20260923 17:23:25.143554 10081 ts_tablet_manager.cc:1397] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260923 17:23:25.143774 10082 tablet_bootstrap.cc:492] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: No bootstrap required, opened a new log
I20260923 17:23:25.143963 10082 ts_tablet_manager.cc:1397] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:23:25.144186 10083 tablet_bootstrap.cc:492] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: No bootstrap required, opened a new log
I20260923 17:23:25.144369 10083 ts_tablet_manager.cc:1397] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:23:25.147109 10082 raft_consensus.cc:348] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.147356 10082 raft_consensus.cc:374] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:25.147409 10082 raft_consensus.cc:729] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d145ce7c987e4e88a480d091bf7fc2df, State: Initialized, Role: FOLLOWER
I20260923 17:23:25.147511 10081 raft_consensus.cc:348] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.147768 10081 raft_consensus.cc:374] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:25.147840 10081 raft_consensus.cc:729] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d612684f67374fc1ad53215f121ee90c, State: Initialized, Role: FOLLOWER
I20260923 17:23:25.147904 10082 consensus_queue.cc:260] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.148283 10081 consensus_queue.cc:260] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.148612 10063 heartbeater.cc:503] Master 127.23.214.190:44497 was elected leader, sending a full tablet report...
I20260923 17:23:25.148932 10082 ts_tablet_manager.cc:1428] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:23:25.149222 10081 ts_tablet_manager.cc:1428] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:23:25.149351 10083 raft_consensus.cc:348] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.149571 9799 heartbeater.cc:503] Master 127.23.214.190:44497 was elected leader, sending a full tablet report...
I20260923 17:23:25.149580 10083 raft_consensus.cc:374] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:25.149650 10083 raft_consensus.cc:729] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965a5aa577f248719e0579079028b51e, State: Initialized, Role: FOLLOWER
I20260923 17:23:25.150066 10083 consensus_queue.cc:260] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.150789 9931 heartbeater.cc:503] Master 127.23.214.190:44497 was elected leader, sending a full tablet report...
I20260923 17:23:25.151036 10083 ts_tablet_manager.cc:1428] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260923 17:23:25.207042 9800 tablet.cc:2369] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:23:25.333395 10064 tablet.cc:2369] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:25.348397 10088 raft_consensus.cc:482] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:25.348615 10088 raft_consensus.cc:504] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.349910 10088 leader_election.cc:288] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 965a5aa577f248719e0579079028b51e (127.23.214.130:45609), d145ce7c987e4e88a480d091bf7fc2df (127.23.214.131:37025)
I20260923 17:23:25.354164 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.350329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:50549 (local address 127.23.214.131:37025)
0923 17:23:25.350542 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.350548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.350664 (+ 116us) server_negotiation.cc:407] Connection header received
0923 17:23:25.350797 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.350801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.350863 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.350990 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:25.351422 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.352100 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.353146 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.353534 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.353696 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.353751 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.353974 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.354041 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 17:23:25.354187 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.350228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:37025 (local address 127.23.214.129:50549)
0923 17:23:25.350542 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:25.350558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:25.350689 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:25.351006 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:25.351009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:25.351030 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:25.351278 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.351287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.352212 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.352217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.353021 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.353029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.353687 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.353690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.353785 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.353850 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 17:23:25.353959 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:23:25.354031 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":55,"thread_start_us":116,"threads_started":1}
I20260923 17:23:25.354358 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.350365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:60895 (local address 127.23.214.130:45609)
0923 17:23:25.350626 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.350633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.350691 (+ 58us) server_negotiation.cc:407] Connection header received
0923 17:23:25.350856 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.350859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.350913 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.350983 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:25.351490 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.352158 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.353205 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.353608 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.353798 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.353860 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.354109 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.354181 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260923 17:23:25.354737 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.350280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:45609 (local address 127.23.214.129:60895)
0923 17:23:25.350613 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:25.350626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:25.350751 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:25.351108 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:25.351110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:25.351129 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:25.351304 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.351310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.352285 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.352291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.353060 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:25.353068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.353790 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.353794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:25.353963 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.353983 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 17:23:25.354099 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:23:25.354173 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":31,"thread_start_us":93,"threads_started":1}
I20260923 17:23:25.354975 10017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e3aacbc68e47f290c75a69d378bc06" candidate_uuid: "d612684f67374fc1ad53215f121ee90c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d145ce7c987e4e88a480d091bf7fc2df" is_pre_election: true
I20260923 17:23:25.355402 10017 raft_consensus.cc:2454] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d612684f67374fc1ad53215f121ee90c in term 0.
I20260923 17:23:25.355472 9885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e3aacbc68e47f290c75a69d378bc06" candidate_uuid: "d612684f67374fc1ad53215f121ee90c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965a5aa577f248719e0579079028b51e" is_pre_election: true
I20260923 17:23:25.355788 9885 raft_consensus.cc:2454] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d612684f67374fc1ad53215f121ee90c in term 0.
I20260923 17:23:25.356019 9687 leader_election.cc:302] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d145ce7c987e4e88a480d091bf7fc2df, d612684f67374fc1ad53215f121ee90c; no voters:
I20260923 17:23:25.356340 10088 raft_consensus.cc:2781] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:23:25.356416 10088 raft_consensus.cc:482] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:23:25.356446 10088 raft_consensus.cc:3037] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:25.357445 10088 raft_consensus.cc:504] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.357852 10088 leader_election.cc:288] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [CANDIDATE]: Term 1 election: Requested vote from peers 965a5aa577f248719e0579079028b51e (127.23.214.130:45609), d145ce7c987e4e88a480d091bf7fc2df (127.23.214.131:37025)
I20260923 17:23:25.358243 9885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e3aacbc68e47f290c75a69d378bc06" candidate_uuid: "d612684f67374fc1ad53215f121ee90c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965a5aa577f248719e0579079028b51e"
I20260923 17:23:25.358399 9885 raft_consensus.cc:3037] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:25.358561 10017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e3aacbc68e47f290c75a69d378bc06" candidate_uuid: "d612684f67374fc1ad53215f121ee90c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d145ce7c987e4e88a480d091bf7fc2df"
I20260923 17:23:25.358685 10017 raft_consensus.cc:3037] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:25.359490 9885 raft_consensus.cc:2454] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d612684f67374fc1ad53215f121ee90c in term 1.
I20260923 17:23:25.359813 9685 leader_election.cc:302] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965a5aa577f248719e0579079028b51e, d612684f67374fc1ad53215f121ee90c; no voters:
I20260923 17:23:25.359915 10017 raft_consensus.cc:2454] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d612684f67374fc1ad53215f121ee90c in term 1.
I20260923 17:23:25.360013 10088 raft_consensus.cc:2781] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:25.360257 10088 raft_consensus.cc:686] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [term 1 LEADER]: Becoming Leader. State: Replica: d612684f67374fc1ad53215f121ee90c, State: Running, Role: LEADER
I20260923 17:23:25.360581 10088 consensus_queue.cc:237] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } }
I20260923 17:23:25.363739 9609 catalog_manager.cc:5495] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c reported cstate change: term changed from 0 to 1, leader changed from <none> to d612684f67374fc1ad53215f121ee90c (127.23.214.129). New cstate: current_term: 1 leader_uuid: "d612684f67374fc1ad53215f121ee90c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d612684f67374fc1ad53215f121ee90c" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 } health_report { overall_health: UNKNOWN } } }
W20260923 17:23:25.388398 9932 tablet.cc:2369] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:25.417271 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.413512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.23.214.129:37289)
0923 17:23:25.413764 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:25.413769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:25.413783 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:23:25.413996 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:25.413998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:25.414058 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:25.414177 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:23:25.414424 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.414944 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.415818 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:25.416039 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:25.416155 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:25.416469 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:23:25.416889 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:23:25.417026 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:25.417067 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 17:23:25.417124 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260923 17:23:25.423303 9885 raft_consensus.cc:1261] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 1 FOLLOWER]: Refusing update from remote peer d612684f67374fc1ad53215f121ee90c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:25.423296 10017 raft_consensus.cc:1261] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 1 FOLLOWER]: Refusing update from remote peer d612684f67374fc1ad53215f121ee90c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:25.424036 10088 consensus_queue.cc:1035] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d145ce7c987e4e88a480d091bf7fc2df" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:25.424232 10092 consensus_queue.cc:1035] T e5e3aacbc68e47f290c75a69d378bc06 P d612684f67374fc1ad53215f121ee90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "965a5aa577f248719e0579079028b51e" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 45609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:25.431063 10095 mvcc.cc:205] Tried to move back new op lower bound from 7332594505406349312 to 7332594505159417856. Current Snapshot: MvccSnapshot[applied={T|T < 7332594505406349312}]
I20260923 17:23:25.433429 10097 mvcc.cc:205] Tried to move back new op lower bound from 7332594505406349312 to 7332594505159417856. Current Snapshot: MvccSnapshot[applied={T|T < 7332594505406349312}]
I20260923 17:23:25.442286 9610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:25.442523 9610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:25.445905 9610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aa8a0265c44400eae09a9cc31b9b9e5: Sending DeleteTablet for 3 replicas of tablet e5e3aacbc68e47f290c75a69d378bc06
I20260923 17:23:25.446707 9865 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e3aacbc68e47f290c75a69d378bc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:25 UTC) from {username='slave'} at 127.0.0.1:56448
I20260923 17:23:25.446821 9997 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e3aacbc68e47f290c75a69d378bc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:25 UTC) from {username='slave'} at 127.0.0.1:47674
I20260923 17:23:25.447278 10104 tablet_replica.cc:330] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: stopping tablet replica
I20260923 17:23:25.447307 9732 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e3aacbc68e47f290c75a69d378bc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:25 UTC) from {username='slave'} at 127.0.0.1:56650
I20260923 17:23:25.447649 10104 raft_consensus.cc:2229] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:25.447750 10105 tablet_replica.cc:330] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: stopping tablet replica
I20260923 17:23:25.447906 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9670
I20260923 17:23:25.447986 10104 raft_consensus.cc:2258] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:25.448043 10105 raft_consensus.cc:2229] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:25.448555 10105 raft_consensus.cc:2258] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:25.450170 10105 ts_tablet_manager.cc:1905] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:25.450331 10104 ts_tablet_manager.cc:1905] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:25.452848 10105 ts_tablet_manager.cc:1918] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:25.452847 10104 ts_tablet_manager.cc:1918] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:25.452950 10105 log.cc:1196] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/e5e3aacbc68e47f290c75a69d378bc06
I20260923 17:23:25.452950 10104 log.cc:1196] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/e5e3aacbc68e47f290c75a69d378bc06
I20260923 17:23:25.453265 10105 ts_tablet_manager.cc:1939] T e5e3aacbc68e47f290c75a69d378bc06 P d145ce7c987e4e88a480d091bf7fc2df: Deleting consensus metadata
I20260923 17:23:25.453265 10104 ts_tablet_manager.cc:1939] T e5e3aacbc68e47f290c75a69d378bc06 P 965a5aa577f248719e0579079028b51e: Deleting consensus metadata
I20260923 17:23:25.454888 9594 catalog_manager.cc:4841] TS 965a5aa577f248719e0579079028b51e (127.23.214.130:45609): tablet e5e3aacbc68e47f290c75a69d378bc06 (table test-table [id=ba131cd5a20d43818874e9cb57b237a5]) successfully deleted
I20260923 17:23:25.454988 9596 catalog_manager.cc:4841] TS d145ce7c987e4e88a480d091bf7fc2df (127.23.214.131:37025): tablet e5e3aacbc68e47f290c75a69d378bc06 (table test-table [id=ba131cd5a20d43818874e9cb57b237a5]) successfully deleted
W20260923 17:23:25.455581 9596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:37289 (error 108)
I20260923 17:23:25.455863 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9802
I20260923 17:23:25.456884 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:25.456578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37289 (local address 127.0.0.1:56676)
0923 17:23:25.456737 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:25.456812 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:23:25.457157 9596 catalog_manager.cc:4584] TS d612684f67374fc1ad53215f121ee90c (127.23.214.129:37289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5e3aacbc68e47f290c75a69d378bc06: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37289: connect: Connection refused (error 111)
I20260923 17:23:25.462793 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9934
I20260923 17:23:25.469341 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 9579
2026-09-23T17:23:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1215 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:25 dist-test-slave-9q7f krb5kdc[10110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:25 dist-test-slave-9q7f krb5kdc[10110](info): set up 2 sockets
Sep 23 17:23:25 dist-test-slave-9q7f krb5kdc[10110](info): commencing operation
krb5kdc: starting...
W20260923 17:23:26.507952 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:26 dist-test-slave-9q7f krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:26.638587 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:36859
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:36859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:26.748463 10123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:26.748739 10123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:26.748780 10123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:26.752925 10123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:26.753008 10123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:26.753028 10123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:26.753046 10123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:26.753060 10123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:26.758731 10123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:36859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:36859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:26.759866 10123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:26.760905 10123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:26.769865 10128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:26.769917 10129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:26.769836 10131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:26.770139 10123 server_base.cc:1034] running on GCE node
I20260923 17:23:26.770871 10123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:26.772017 10123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:26.773208 10123 hybrid_clock.cc:648] HybridClock initialized: now 1790184206773193 us; error 28 us; skew 500 ppm
Sep 23 17:23:26 dist-test-slave-9q7f krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184206, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:26.776011 10123 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:26.776842 10123 webserver.cc:460] Webserver started at http://127.23.214.190:34399/ using document root <none> and password file <none>
I20260923 17:23:26.777357 10123 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:26.777413 10123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:26.777575 10123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:26.779634 10123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "f3bf1baeb2354ba9ac3160071fc7a2df"
format_stamp: "Formatted at 2026-09-23 17:23:26 on dist-test-slave-9q7f"
server_key: "bd21222b64c1ec81d5dd3c701d24b311"
server_key_iv: "96c952335b87cc0e83ed6b65c17ae718"
server_key_version: "encryptionkey@0"
I20260923 17:23:26.780143 10123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "f3bf1baeb2354ba9ac3160071fc7a2df"
format_stamp: "Formatted at 2026-09-23 17:23:26 on dist-test-slave-9q7f"
server_key: "bd21222b64c1ec81d5dd3c701d24b311"
server_key_iv: "96c952335b87cc0e83ed6b65c17ae718"
server_key_version: "encryptionkey@0"
I20260923 17:23:26.783780 10123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:23:26.786190 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:26.787434 10123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:23:26.787560 10123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "f3bf1baeb2354ba9ac3160071fc7a2df"
format_stamp: "Formatted at 2026-09-23 17:23:26 on dist-test-slave-9q7f"
server_key: "bd21222b64c1ec81d5dd3c701d24b311"
server_key_iv: "96c952335b87cc0e83ed6b65c17ae718"
server_key_version: "encryptionkey@0"
I20260923 17:23:26.787668 10123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:26.804893 10123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:26.807835 10123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:26.808044 10123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:26.814826 10123 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:36859
I20260923 17:23:26.814867 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:36859 every 8 connection(s)
I20260923 17:23:26.815817 10123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:26.818727 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:26.824450 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df: Bootstrap starting.
I20260923 17:23:26.825214 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 10123
I20260923 17:23:26.825366 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:26.825700 24410 external_mini_cluster.cc:1428] Setting key 970b08014eebc6abfff7165a370e993b
I20260923 17:23:26.827613 10191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:26.828347 10191 log.cc:824] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:26.830567 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df: No bootstrap required, opened a new log
Sep 23 17:23:26 dist-test-slave-9q7f krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:26.835142 10191 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 36859 } }
I20260923 17:23:26.835391 10191 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:26.835470 10191 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bf1baeb2354ba9ac3160071fc7a2df, State: Initialized, Role: FOLLOWER
I20260923 17:23:26.835945 10191 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 36859 } }
I20260923 17:23:26.836088 10191 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:26.836158 10191 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:26.836292 10191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:26.837167 10191 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 36859 } }
I20260923 17:23:26.837503 10191 leader_election.cc:302] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3bf1baeb2354ba9ac3160071fc7a2df; no voters:
I20260923 17:23:26.837850 10191 leader_election.cc:288] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:26.837987 10196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:26.838239 10196 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [term 1 LEADER]: Becoming Leader. State: Replica: f3bf1baeb2354ba9ac3160071fc7a2df, State: Running, Role: LEADER
I20260923 17:23:26.838600 10196 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 36859 } }
I20260923 17:23:26.839170 10191 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:26.839859 10196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3bf1baeb2354ba9ac3160071fc7a2df. Latest consensus state: current_term: 1 leader_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 36859 } } }
I20260923 17:23:26.840072 10196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:26.840250 10197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf1baeb2354ba9ac3160071fc7a2df" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 36859 } } }
I20260923 17:23:26.840358 10197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:26.841288 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:26.826963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.23.214.190:36859)
0923 17:23:26.827530 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:26.827545 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:26.827578 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:23:26.828269 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:26.828287 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:26.828682 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:26.829066 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:26.829982 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:26.831019 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:26.832181 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:26.832663 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:26.832787 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:26.835673 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:26.835678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:26.835691 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:26.835718 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:26.837635 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:26.838245 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:26.838251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:26.838269 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:26.838403 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:26.838721 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:26.838724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:26.838726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:26.839130 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:26.839301 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:26.839734 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 17:23:26.840071 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":264,"threads_started":1}
W20260923 17:23:26.845005 10211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:23:26.845089 10211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:23:26.845166 10205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:26.846220 10205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:26.853168 10205 catalog_manager.cc:1347] Generated new cluster ID: fd59182715d0408489f7740159b01d5c
I20260923 17:23:26.853240 10205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:26.877919 10205 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:26.879207 10205 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:26.885524 10205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df: Generated new TSK 0
I20260923 17:23:26.886359 10205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:23:26.918759 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:36859
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:27.033536 10217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:27.033823 10217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:27.033923 10217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:27.039927 10217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:27.040019 10217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:27.040118 10217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:23:27.045816 10217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:27.046986 10217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:27.048026 10217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:27.057941 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:27.057973 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:27.057973 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:27.058327 10217 server_base.cc:1034] running on GCE node
I20260923 17:23:27.058683 10217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:27.059268 10217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:27.060462 10217 hybrid_clock.cc:648] HybridClock initialized: now 1790184207060443 us; error 37 us; skew 500 ppm
Sep 23 17:23:27 dist-test-slave-9q7f krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184207, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:27.063344 10217 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:23:27.064307 10217 webserver.cc:460] Webserver started at http://127.23.214.129:37379/ using document root <none> and password file <none>
I20260923 17:23:27.064906 10217 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:27.064971 10217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:27.065212 10217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:27.067394 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "10250245e2834ee8bd16916345a50383"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "d4c64d9c76c721bc8517baad7ec655b4"
server_key_iv: "9f9a0d44f82e0abe1877c1a71fd87c16"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.067929 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "10250245e2834ee8bd16916345a50383"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "d4c64d9c76c721bc8517baad7ec655b4"
server_key_iv: "9f9a0d44f82e0abe1877c1a71fd87c16"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.071753 10217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:23:27.074103 10232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.075400 10217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:27.075498 10217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "10250245e2834ee8bd16916345a50383"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "d4c64d9c76c721bc8517baad7ec655b4"
server_key_iv: "9f9a0d44f82e0abe1877c1a71fd87c16"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.075585 10217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:27.102314 10217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:27.105644 10217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:27.105839 10217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:27.106443 10217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:27.107203 10217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:27.107247 10217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.107295 10217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:27.107316 10217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.115984 10217 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:37065
I20260923 17:23:27.116019 10347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:37065 every 8 connection(s)
I20260923 17:23:27.116875 10217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:27.126250 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 10217
I20260923 17:23:27.126384 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
Sep 23 17:23:27 dist-test-slave-9q7f krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184207, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:27.126761 24410 external_mini_cluster.cc:1428] Setting key feec67b65ced0b96af3d908754ec7f9e
I20260923 17:23:27.131558 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.118805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:32903 (local address 127.23.214.190:36859)
0923 17:23:27.118956 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.118960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.119814 (+ 854us) server_negotiation.cc:407] Connection header received
0923 17:23:27.121133 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.121136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.121192 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.121322 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:27.122530 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.123068 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.124214 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.124471 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.124587 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.127415 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:27.127418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:27.127421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:27.127453 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:27.129304 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.129817 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.129823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.129828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.129969 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.130414 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.130417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.130418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.130589 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:27.130667 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.131316 (+ 649us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.131434 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:23:27.132829 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.119226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:36859 (local address 127.23.214.129:32903)
0923 17:23:27.119663 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.119705 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.120917 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.121476 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.121483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.121862 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:27.122352 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.122365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.123197 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.123202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.124088 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.124097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.124633 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.124638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.124808 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.125391 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:27.125407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:27.127237 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:27.129430 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:27.129435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:27.129448 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:27.129714 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:27.130073 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:27.130077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:27.130079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:27.130301 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:27.130691 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:27.130700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:27.131045 (+ 345us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.131303 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.131697 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260923 17:23:27.134279 10348 heartbeater.cc:347] Connected to a master server at 127.23.214.190:36859
I20260923 17:23:27.134543 10348 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:27.135066 10348 heartbeater.cc:511] Master 127.23.214.190:36859 requested a full tablet report, sending...
I20260923 17:23:27.136844 10155 ts_manager.cc:194] Registered new tserver with Master: 10250245e2834ee8bd16916345a50383 (127.23.214.129:37065)
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:23:27.138293 10155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:32903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:23:27.155210 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:36859
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:27.267234 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:27.267494 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:27.267562 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:27.271540 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:27.271590 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:27.271703 10353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:23:27.276902 10353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:27.277961 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:27.279022 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:27.288823 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:27.288831 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:27.288811 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:27.289099 10353 server_base.cc:1034] running on GCE node
I20260923 17:23:27.289602 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:27.290334 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:27.291548 10353 hybrid_clock.cc:648] HybridClock initialized: now 1790184207291517 us; error 41 us; skew 500 ppm
Sep 23 17:23:27 dist-test-slave-9q7f krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184207, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:27.294411 10353 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:23:27.295284 10353 webserver.cc:460] Webserver started at http://127.23.214.130:45133/ using document root <none> and password file <none>
I20260923 17:23:27.295878 10353 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:27.295965 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:27.296190 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:27.298230 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "26f887c5e9434114bd2825746da3d40f"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "938a93d5b1a0a75cb7e2d5d2adf978d8"
server_key_iv: "7563bfb3c62dc57bc6a0a6dc2e14436b"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.298764 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "26f887c5e9434114bd2825746da3d40f"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "938a93d5b1a0a75cb7e2d5d2adf978d8"
server_key_iv: "7563bfb3c62dc57bc6a0a6dc2e14436b"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.302417 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:27.304781 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.306015 10353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:27.306162 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "26f887c5e9434114bd2825746da3d40f"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "938a93d5b1a0a75cb7e2d5d2adf978d8"
server_key_iv: "7563bfb3c62dc57bc6a0a6dc2e14436b"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.306279 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:27.317698 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:27.320839 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:27.321029 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:27.321609 10353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:27.322439 10353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:27.322511 10353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.322576 10353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:27.322625 10353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.331542 10353 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:40613
I20260923 17:23:27.331579 10482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:40613 every 8 connection(s)
I20260923 17:23:27.332392 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:27.341800 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 10353
I20260923 17:23:27.341988 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:23:27.342355 24410 external_mini_cluster.cc:1428] Setting key b9a0b9ff9b8a8d769dc8fff887d352f2
Sep 23 17:23:27 dist-test-slave-9q7f krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184207, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:27.346868 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.334079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:45857 (local address 127.23.214.190:36859)
0923 17:23:27.334215 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.334219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.334994 (+ 775us) server_negotiation.cc:407] Connection header received
0923 17:23:27.336365 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.336369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.336425 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.336506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:27.337591 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.338203 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.339834 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.340107 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.340234 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.343189 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:27.343192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:27.343193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:27.343217 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:27.344745 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.345290 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.345294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.345295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.345377 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.345718 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.345722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.345724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.345978 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:27.346079 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.346585 (+ 506us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.346712 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:23:27.347662 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.334428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:36859 (local address 127.23.214.130:45857)
0923 17:23:27.334852 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.334892 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.336152 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.336663 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.336669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.336997 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:27.337450 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.337462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.338337 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.338345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.339667 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.339678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.340271 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.340276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.340472 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.341292 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:27.341310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:27.343014 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:27.344883 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:27.344888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:27.344904 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:27.345185 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:27.345471 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:27.345476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:27.345478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:27.345620 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:27.346089 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:27.346104 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:27.346355 (+ 251us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.346545 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.346867 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260923 17:23:27.349011 10483 heartbeater.cc:347] Connected to a master server at 127.23.214.190:36859
I20260923 17:23:27.349301 10483 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:27.349910 10483 heartbeater.cc:511] Master 127.23.214.190:36859 requested a full tablet report, sending...
I20260923 17:23:27.351084 10155 ts_manager.cc:194] Registered new tserver with Master: 26f887c5e9434114bd2825746da3d40f (127.23.214.130:40613)
I20260923 17:23:27.351691 10155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:45857
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:23:27.369143 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:36859
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:27.483572 10488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:27.483831 10488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:27.483873 10488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:27.487934 10488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:27.487993 10488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:27.488049 10488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:23:27.493341 10488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:27.494354 10488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:27.495406 10488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:27.504951 10496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:27.504964 10494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:27.504959 10493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:27.505260 10488 server_base.cc:1034] running on GCE node
I20260923 17:23:27.505648 10488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:27.506338 10488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:27.507519 10488 hybrid_clock.cc:648] HybridClock initialized: now 1790184207507490 us; error 50 us; skew 500 ppm
Sep 23 17:23:27 dist-test-slave-9q7f krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184207, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:27.510473 10488 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:23:27.511276 10488 webserver.cc:460] Webserver started at http://127.23.214.131:40883/ using document root <none> and password file <none>
I20260923 17:23:27.511806 10488 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:27.511864 10488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:27.512025 10488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:27.514061 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "32b12263cf4b463bad5c7b5d342818db"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "503701612515b163d2ee46c99bc1455e"
server_key_iv: "6020f661c68bbaf05aa0c5528b2a65a3"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.514571 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "32b12263cf4b463bad5c7b5d342818db"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "503701612515b163d2ee46c99bc1455e"
server_key_iv: "6020f661c68bbaf05aa0c5528b2a65a3"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.518293 10488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:27.520576 10503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.521817 10488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:27.521972 10488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "32b12263cf4b463bad5c7b5d342818db"
format_stamp: "Formatted at 2026-09-23 17:23:27 on dist-test-slave-9q7f"
server_key: "503701612515b163d2ee46c99bc1455e"
server_key_iv: "6020f661c68bbaf05aa0c5528b2a65a3"
server_key_version: "encryptionkey@0"
I20260923 17:23:27.522079 10488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:27.532758 10488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:27.536281 10488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:27.536491 10488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:27.537087 10488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:27.537981 10488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:27.538033 10488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.538108 10488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:27.538152 10488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:27.547576 10488 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:45141
I20260923 17:23:27.547603 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:45141 every 8 connection(s)
I20260923 17:23:27.548475 10488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:27.555563 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 10488
I20260923 17:23:27.555735 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:23:27.556054 24410 external_mini_cluster.cc:1428] Setting key 7a1d2b4b0f3f9b49f8c46ce3b1eb6f74
Sep 23 17:23:27 dist-test-slave-9q7f krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184207, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:27.562278 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.550330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:60275 (local address 127.23.214.190:36859)
0923 17:23:27.550495 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.550500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.551398 (+ 898us) server_negotiation.cc:407] Connection header received
0923 17:23:27.552637 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.552641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.552688 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.552763 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:27.553982 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.554525 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.555646 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.555965 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.556081 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.558778 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:27.558781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:27.558783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:27.558808 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:27.560271 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.560820 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.560823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.560825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.560899 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.561205 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.561207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.561208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.561379 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:27.561454 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.562051 (+ 597us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.562150 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:23:27.563179 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.550784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:36859 (local address 127.23.214.131:60275)
0923 17:23:27.551233 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.551284 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.552438 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.552932 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.552939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.553283 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:27.553788 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.553802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.554639 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.554642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.555511 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.555520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.556128 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.556133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.556310 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.556952 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:27.556968 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:27.558625 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:27.560411 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:27.560416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:27.560429 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:27.560728 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:27.560982 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:27.560984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:27.560985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:27.561127 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:27.561500 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:27.561505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:27.561776 (+ 271us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.562019 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.562292 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260923 17:23:27.564302 10618 heartbeater.cc:347] Connected to a master server at 127.23.214.190:36859
I20260923 17:23:27.564560 10618 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:27.565054 10618 heartbeater.cc:511] Master 127.23.214.190:36859 requested a full tablet report, sending...
I20260923 17:23:27.566145 10155 ts_manager.cc:194] Registered new tserver with Master: 32b12263cf4b463bad5c7b5d342818db (127.23.214.131:45141)
I20260923 17:23:27.566733 10155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:60275
I20260923 17:23:27.569964 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:23:27.579814 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.571973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.23.214.190:36859)
0923 17:23:27.572095 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.572100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.572303 (+ 203us) server_negotiation.cc:407] Connection header received
0923 17:23:27.572390 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.572392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.572428 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.572492 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:27.573208 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.573702 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.574575 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.574783 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.574860 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.576332 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:27.576335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:27.576337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:27.576358 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:27.577839 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.578422 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.578426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.578427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.578509 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:27.578836 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:27.578839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:27.578841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:27.579118 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:27.579304 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.579554 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.579645 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:23:27.582827 10155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:23:27.585989 10155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:23:27.600703 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.594228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.23.214.131:45141)
0923 17:23:27.594647 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.594661 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.595048 (+ 387us) server_negotiation.cc:407] Connection header received
0923 17:23:27.595253 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.595267 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.595777 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.595900 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:27.596259 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.597481 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.599056 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.599905 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.600039 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.600191 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.600416 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.600521 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":86,"threads_started":1}
I20260923 17:23:27.600765 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.594883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.23.214.130:40613)
0923 17:23:27.595320 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.595331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.595574 (+ 243us) server_negotiation.cc:407] Connection header received
0923 17:23:27.595685 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.595690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.595908 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.596055 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:27.596297 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.597481 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.599265 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.600041 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.600160 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.600269 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.600516 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.600610 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":125,"threads_started":1}
I20260923 17:23:27.600756 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.594187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40613 (local address 127.0.0.1:39872)
0923 17:23:27.595455 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.595470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.595568 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.596053 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.596056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.596074 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:27.596173 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.596181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.597629 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.597634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.599133 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.599144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.600269 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.600274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.600366 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.600383 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.600480 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.600603 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":41,"thread_start_us":109,"threads_started":1}
I20260923 17:23:27.601106 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.594133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:45141 (local address 127.0.0.1:52180)
0923 17:23:27.594890 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.594941 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.595083 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.595921 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.595928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.595950 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:27.596103 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.596111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.597682 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.597685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.598920 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.598930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.600036 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.600040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.600164 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.600221 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.600913 (+ 692us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.600982 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":118,"thread_start_us":71,"threads_started":1}
I20260923 17:23:27.602321 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.595212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.23.214.129:37065)
0923 17:23:27.596160 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.596174 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.596196 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:23:27.596282 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.596287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.596446 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.596619 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:27.596848 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.597616 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.599488 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.600125 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.600243 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.600380 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.602076 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.602189 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":125,"threads_started":1}
I20260923 17:23:27.602530 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.594802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37065 (local address 127.0.0.1:56714)
0923 17:23:27.595547 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.595562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.595689 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.596597 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.596600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.596617 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:27.596713 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.596719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.598186 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.598189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.599380 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.599387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.600439 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.600443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.600538 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.600560 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.600626 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.600687 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":81,"threads_started":1}
I20260923 17:23:27.603871 10416 tablet_service.cc:1467] Processing CreateTablet for tablet 76e00ef5a00f45e8868f08f6800513ae (DEFAULT_TABLE table=test-table [id=b10bc8c8df0c4f38bfdbcdb838f16200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:27.603875 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 76e00ef5a00f45e8868f08f6800513ae (DEFAULT_TABLE table=test-table [id=b10bc8c8df0c4f38bfdbcdb838f16200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:27.604810 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e00ef5a00f45e8868f08f6800513ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:27.604841 10416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e00ef5a00f45e8868f08f6800513ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:27.606913 10281 tablet_service.cc:1467] Processing CreateTablet for tablet 76e00ef5a00f45e8868f08f6800513ae (DEFAULT_TABLE table=test-table [id=b10bc8c8df0c4f38bfdbcdb838f16200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:27.607986 10281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e00ef5a00f45e8868f08f6800513ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:27.611451 10635 tablet_bootstrap.cc:492] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Bootstrap starting.
I20260923 17:23:27.612799 10636 tablet_bootstrap.cc:492] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Bootstrap starting.
I20260923 17:23:27.614122 10635 tablet_bootstrap.cc:654] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:27.614921 10636 tablet_bootstrap.cc:654] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:27.615029 10637 tablet_bootstrap.cc:492] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383: Bootstrap starting.
I20260923 17:23:27.615450 10635 log.cc:824] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:27.616125 10636 log.cc:824] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:27.617002 10637 tablet_bootstrap.cc:654] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:27.618047 10635 tablet_bootstrap.cc:492] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: No bootstrap required, opened a new log
I20260923 17:23:27.618204 10636 tablet_bootstrap.cc:492] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: No bootstrap required, opened a new log
I20260923 17:23:27.618229 10637 log.cc:824] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:27.618278 10635 ts_tablet_manager.cc:1397] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:23:27.618402 10636 ts_tablet_manager.cc:1397] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260923 17:23:27.620154 10637 tablet_bootstrap.cc:492] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383: No bootstrap required, opened a new log
I20260923 17:23:27.620338 10637 ts_tablet_manager.cc:1397] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:23:27.621445 10635 raft_consensus.cc:348] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.621707 10635 raft_consensus.cc:374] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:27.621762 10635 raft_consensus.cc:729] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f887c5e9434114bd2825746da3d40f, State: Initialized, Role: FOLLOWER
I20260923 17:23:27.622241 10635 consensus_queue.cc:260] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.622146 10636 raft_consensus.cc:348] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.622429 10636 raft_consensus.cc:374] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:27.622479 10636 raft_consensus.cc:729] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b12263cf4b463bad5c7b5d342818db, State: Initialized, Role: FOLLOWER
I20260923 17:23:27.622938 10483 heartbeater.cc:503] Master 127.23.214.190:36859 was elected leader, sending a full tablet report...
I20260923 17:23:27.622912 10636 consensus_queue.cc:260] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.623934 10636 ts_tablet_manager.cc:1428] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260923 17:23:27.624279 10618 heartbeater.cc:503] Master 127.23.214.190:36859 was elected leader, sending a full tablet report...
I20260923 17:23:27.624155 10637 raft_consensus.cc:348] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.624396 10637 raft_consensus.cc:374] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:27.624465 10637 raft_consensus.cc:729] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10250245e2834ee8bd16916345a50383, State: Initialized, Role: FOLLOWER
I20260923 17:23:27.624864 10637 consensus_queue.cc:260] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.625511 10348 heartbeater.cc:503] Master 127.23.214.190:36859 was elected leader, sending a full tablet report...
I20260923 17:23:27.625919 10637 ts_tablet_manager.cc:1428] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:23:27.626557 10635 ts_tablet_manager.cc:1428] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260923 17:23:27.799459 10619 tablet.cc:2369] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:27.812726 10643 raft_consensus.cc:482] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:27.812975 10643 raft_consensus.cc:504] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.814240 10643 leader_election.cc:288] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32b12263cf4b463bad5c7b5d342818db (127.23.214.131:45141), 26f887c5e9434114bd2825746da3d40f (127.23.214.130:40613)
I20260923 17:23:27.819092 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.814589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:42271 (local address 127.23.214.131:45141)
0923 17:23:27.814772 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.814778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.815038 (+ 260us) server_negotiation.cc:407] Connection header received
0923 17:23:27.815308 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.815311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.815365 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.815439 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:27.815731 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.816230 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.817847 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.818471 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.818588 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.818632 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.818879 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.818953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 17:23:27.819118 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.814968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:47481 (local address 127.23.214.130:40613)
0923 17:23:27.815091 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.815095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.815109 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:23:27.815224 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.815229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.815296 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.815381 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:27.815688 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.816452 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.817948 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.818340 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.818415 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.818454 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.818939 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.819016 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:23:27.819115 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.814561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:40613 (local address 127.23.214.129:47481)
0923 17:23:27.814973 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.814987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.815091 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.815401 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.815404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.815421 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:27.815558 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.815565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.816562 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.816565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.817754 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.817764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.818562 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.818567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.818706 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.818776 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.818851 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.818962 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":66,"thread_start_us":131,"threads_started":1}
I20260923 17:23:27.819485 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.814440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:45141 (local address 127.23.214.129:42271)
0923 17:23:27.814948 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.814963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:27.815115 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:27.815518 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:27.815523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:27.815536 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:27.815635 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.815645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.816350 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.816355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.817707 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:27.817719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.818589 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.818593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:27.818726 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.818771 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 17:23:27.818851 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:23:27.819348 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":492,"thread_start_us":91,"threads_started":1}
I20260923 17:23:27.819923 10436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e00ef5a00f45e8868f08f6800513ae" candidate_uuid: "10250245e2834ee8bd16916345a50383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f887c5e9434114bd2825746da3d40f" is_pre_election: true
I20260923 17:23:27.820065 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e00ef5a00f45e8868f08f6800513ae" candidate_uuid: "10250245e2834ee8bd16916345a50383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b12263cf4b463bad5c7b5d342818db" is_pre_election: true
I20260923 17:23:27.820236 10436 raft_consensus.cc:2454] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10250245e2834ee8bd16916345a50383 in term 0.
I20260923 17:23:27.820382 10571 raft_consensus.cc:2454] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10250245e2834ee8bd16916345a50383 in term 0.
I20260923 17:23:27.820739 10234 leader_election.cc:302] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10250245e2834ee8bd16916345a50383, 26f887c5e9434114bd2825746da3d40f; no voters:
I20260923 17:23:27.821061 10643 raft_consensus.cc:2781] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:23:27.821161 10643 raft_consensus.cc:482] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:23:27.821216 10643 raft_consensus.cc:3037] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:27.822336 10643 raft_consensus.cc:504] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.822731 10643 leader_election.cc:288] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [CANDIDATE]: Term 1 election: Requested vote from peers 32b12263cf4b463bad5c7b5d342818db (127.23.214.131:45141), 26f887c5e9434114bd2825746da3d40f (127.23.214.130:40613)
I20260923 17:23:27.823060 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e00ef5a00f45e8868f08f6800513ae" candidate_uuid: "10250245e2834ee8bd16916345a50383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b12263cf4b463bad5c7b5d342818db"
I20260923 17:23:27.823182 10571 raft_consensus.cc:3037] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:27.824447 10571 raft_consensus.cc:2454] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10250245e2834ee8bd16916345a50383 in term 1.
I20260923 17:23:27.824699 10436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e00ef5a00f45e8868f08f6800513ae" candidate_uuid: "10250245e2834ee8bd16916345a50383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f887c5e9434114bd2825746da3d40f"
I20260923 17:23:27.824769 10237 leader_election.cc:302] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10250245e2834ee8bd16916345a50383, 32b12263cf4b463bad5c7b5d342818db; no voters:
I20260923 17:23:27.824824 10436 raft_consensus.cc:3037] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:27.824961 10643 raft_consensus.cc:2781] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:27.825201 10643 raft_consensus.cc:686] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [term 1 LEADER]: Becoming Leader. State: Replica: 10250245e2834ee8bd16916345a50383, State: Running, Role: LEADER
I20260923 17:23:27.825592 10643 consensus_queue.cc:237] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } }
I20260923 17:23:27.826138 10436 raft_consensus.cc:2454] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10250245e2834ee8bd16916345a50383 in term 1.
I20260923 17:23:27.829627 10154 catalog_manager.cc:5495] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10250245e2834ee8bd16916345a50383 (127.23.214.129). New cstate: current_term: 1 leader_uuid: "10250245e2834ee8bd16916345a50383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10250245e2834ee8bd16916345a50383" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37065 } health_report { overall_health: HEALTHY } } }
W20260923 17:23:27.833438 10484 tablet.cc:2369] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:23:27.868185 10349 tablet.cc:2369] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:27.893196 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.889442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.23.214.129:37065)
0923 17:23:27.889584 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:27.889589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:27.889607 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:23:27.889700 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:27.889702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:27.889771 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:27.889912 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:23:27.890174 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.890717 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.891638 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:27.891864 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:27.891975 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:27.892322 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:23:27.892735 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:23:27.892907 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:27.893014 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 17:23:27.893060 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:23:27.898984 10571 raft_consensus.cc:1261] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 1 FOLLOWER]: Refusing update from remote peer 10250245e2834ee8bd16916345a50383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:27.898983 10436 raft_consensus.cc:1261] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 1 FOLLOWER]: Refusing update from remote peer 10250245e2834ee8bd16916345a50383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:27.899703 10643 consensus_queue.cc:1035] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f887c5e9434114bd2825746da3d40f" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:27.899940 10646 consensus_queue.cc:1035] T 76e00ef5a00f45e8868f08f6800513ae P 10250245e2834ee8bd16916345a50383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b12263cf4b463bad5c7b5d342818db" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:27.907722 10650 mvcc.cc:205] Tried to move back new op lower bound from 7332594515546898432 to 7332594515257966592. Current Snapshot: MvccSnapshot[applied={T|T < 7332594515546898432}]
I20260923 17:23:27.911496 10651 mvcc.cc:205] Tried to move back new op lower bound from 7332594515546898432 to 7332594515257966592. Current Snapshot: MvccSnapshot[applied={T|T < 7332594515546898432 or (T in {7332594515546898432})}]
I20260923 17:23:27.915884 10154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:27.916074 10154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:27.918499 10154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3bf1baeb2354ba9ac3160071fc7a2df: Sending DeleteTablet for 3 replicas of tablet 76e00ef5a00f45e8868f08f6800513ae
I20260923 17:23:27.919293 10416 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e00ef5a00f45e8868f08f6800513ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:27 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:39872
I20260923 17:23:27.919287 10281 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e00ef5a00f45e8868f08f6800513ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:27 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:56714
I20260923 17:23:27.919337 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e00ef5a00f45e8868f08f6800513ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:27 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:52180
I20260923 17:23:27.919822 10660 tablet_replica.cc:330] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: stopping tablet replica
I20260923 17:23:27.920105 10659 tablet_replica.cc:330] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: stopping tablet replica
I20260923 17:23:27.920141 10660 raft_consensus.cc:2229] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:27.920480 10660 raft_consensus.cc:2258] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:27.920475 10659 raft_consensus.cc:2229] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:27.920606 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 10217
I20260923 17:23:27.921416 10659 raft_consensus.cc:2258] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:27.923022 10659 ts_tablet_manager.cc:1905] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:27.923276 10660 ts_tablet_manager.cc:1905] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:27.925446 10660 ts_tablet_manager.cc:1918] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:27.925556 10660 log.cc:1196] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/76e00ef5a00f45e8868f08f6800513ae
I20260923 17:23:27.925848 10660 ts_tablet_manager.cc:1939] T 76e00ef5a00f45e8868f08f6800513ae P 32b12263cf4b463bad5c7b5d342818db: Deleting consensus metadata
I20260923 17:23:27.926460 10659 ts_tablet_manager.cc:1918] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:27.926555 10659 log.cc:1196] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/76e00ef5a00f45e8868f08f6800513ae
I20260923 17:23:27.926857 10659 ts_tablet_manager.cc:1939] T 76e00ef5a00f45e8868f08f6800513ae P 26f887c5e9434114bd2825746da3d40f: Deleting consensus metadata
I20260923 17:23:27.927403 10142 catalog_manager.cc:4841] TS 32b12263cf4b463bad5c7b5d342818db (127.23.214.131:45141): tablet 76e00ef5a00f45e8868f08f6800513ae (table test-table [id=b10bc8c8df0c4f38bfdbcdb838f16200]) successfully deleted
I20260923 17:23:27.928269 10140 catalog_manager.cc:4841] TS 26f887c5e9434114bd2825746da3d40f (127.23.214.130:40613): tablet 76e00ef5a00f45e8868f08f6800513ae (table test-table [id=b10bc8c8df0c4f38bfdbcdb838f16200]) successfully deleted
W20260923 17:23:27.929207 10142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:37065 (error 108)
I20260923 17:23:27.929380 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 10353
I20260923 17:23:27.930845 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:27.930460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37065 (local address 127.0.0.1:56732)
0923 17:23:27.930618 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:27.930739 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:23:27.931010 10142 catalog_manager.cc:4584] TS 10250245e2834ee8bd16916345a50383 (127.23.214.129:37065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76e00ef5a00f45e8868f08f6800513ae: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37065: connect: Connection refused (error 111)
I20260923 17:23:27.936122 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 10488
I20260923 17:23:27.942291 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 10123
2026-09-23T17:23:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2473 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T17:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:27Z Disabled control of system clock
I20260923 17:23:27.972328 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44735
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:44735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:23:28.083309 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:28.083598 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:28.083636 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:28.087741 10665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:28.087807 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:28.087824 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:28.087841 10665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:28.087857 10665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:28.092757 10665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:44735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:28.093952 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:28.095350 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:28.103943 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:28.103960 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:28.104001 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:28.104156 10665 server_base.cc:1034] running on GCE node
I20260923 17:23:28.104772 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:28.105989 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:28.107159 10665 hybrid_clock.cc:648] HybridClock initialized: now 1790184208107138 us; error 34 us; skew 500 ppm
I20260923 17:23:28.109179 10665 webserver.cc:460] Webserver started at http://127.23.214.190:38427/ using document root <none> and password file <none>
I20260923 17:23:28.109776 10665 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:28.109839 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:28.110110 10665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:28.112226 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "fd2b57e183d949ec95ac06685881c674"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "beca49314b4a48bc5d437a4f2028f282"
server_key_iv: "e19aae840989c93f9ec0d2023fbe9bf2"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.112818 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "fd2b57e183d949ec95ac06685881c674"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "beca49314b4a48bc5d437a4f2028f282"
server_key_iv: "e19aae840989c93f9ec0d2023fbe9bf2"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.116775 10665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260923 17:23:28.119304 10679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.120548 10665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:28.120677 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "fd2b57e183d949ec95ac06685881c674"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "beca49314b4a48bc5d437a4f2028f282"
server_key_iv: "e19aae840989c93f9ec0d2023fbe9bf2"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.120795 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:28.134235 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:28.134950 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:28.135161 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:28.142019 10665 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:44735
I20260923 17:23:28.142040 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:44735 every 8 connection(s)
I20260923 17:23:28.142995 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:28.146239 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:28.148105 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 10665
I20260923 17:23:28.148240 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:28.148542 24410 external_mini_cluster.cc:1428] Setting key 94e0631b61606296776950650a02d8a8
I20260923 17:23:28.152949 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674: Bootstrap starting.
I20260923 17:23:28.155252 10732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:28.155987 10732 log.cc:824] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:28.158007 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.150037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.23.214.190:44735)
0923 17:23:28.150684 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.150700 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.150736 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:23:28.151419 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.151436 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.151839 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.152175 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:28.152596 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.153584 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.154533 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.155010 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.155193 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.156067 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:28.156072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:28.156085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:28.156582 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:28.156654 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:28.156682 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:28.156786 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.156948 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.157229 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
I20260923 17:23:28.158031 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674: No bootstrap required, opened a new log
I20260923 17:23:28.161823 10732 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2b57e183d949ec95ac06685881c674" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44735 } }
I20260923 17:23:28.162132 10732 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:28.162212 10732 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2b57e183d949ec95ac06685881c674, State: Initialized, Role: FOLLOWER
I20260923 17:23:28.162703 10732 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2b57e183d949ec95ac06685881c674" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44735 } }
I20260923 17:23:28.162881 10732 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:28.162949 10732 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:28.163048 10732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:28.164000 10732 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2b57e183d949ec95ac06685881c674" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44735 } }
I20260923 17:23:28.164348 10732 leader_election.cc:302] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd2b57e183d949ec95ac06685881c674; no voters:
I20260923 17:23:28.164654 10732 leader_election.cc:288] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:28.164781 10737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:28.165030 10737 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [term 1 LEADER]: Becoming Leader. State: Replica: fd2b57e183d949ec95ac06685881c674, State: Running, Role: LEADER
I20260923 17:23:28.165333 10737 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2b57e183d949ec95ac06685881c674" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44735 } }
I20260923 17:23:28.165982 10732 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:28.167037 10739 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd2b57e183d949ec95ac06685881c674. Latest consensus state: current_term: 1 leader_uuid: "fd2b57e183d949ec95ac06685881c674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2b57e183d949ec95ac06685881c674" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44735 } } }
I20260923 17:23:28.167157 10739 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:28.167529 10746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:28.167557 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd2b57e183d949ec95ac06685881c674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2b57e183d949ec95ac06685881c674" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44735 } } }
I20260923 17:23:28.167810 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:28.170753 10746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:28.176049 10746 catalog_manager.cc:1347] Generated new cluster ID: 5e51eefe01924711a055db216b965dbb
I20260923 17:23:28.176121 10746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:28.189491 10746 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:28.190479 10746 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:28.195374 10746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674: Generated new TSK 0
I20260923 17:23:28.196022 10746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:28.199415 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44735
--builtin_ntp_servers=127.23.214.148:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:23:28.330669 10757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:28.330978 10757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:28.331058 10757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:28.335211 10757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:28.335301 10757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:28.335397 10757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:23:28.340230 10757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:28.341537 10757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:28.342958 10757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:23:28.352895 10757 server_base.cc:1034] running on GCE node
W20260923 17:23:28.352855 10762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:28.352886 10763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:28.352855 10765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:28.353499 10757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:28.354249 10757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:28.355440 10757 hybrid_clock.cc:648] HybridClock initialized: now 1790184208355426 us; error 25 us; skew 500 ppm
I20260923 17:23:28.357283 10757 webserver.cc:460] Webserver started at http://127.23.214.129:44317/ using document root <none> and password file <none>
I20260923 17:23:28.357930 10757 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:28.358026 10757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:28.358266 10757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:28.360303 10757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "018b703e741b4353b4695d6d039d68a9"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "f7bc733173c6b03e1c488b68b78df940"
server_key_iv: "07ee4fccf7d85dd58eb0bf8d3cd358c0"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.360883 10757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "018b703e741b4353b4695d6d039d68a9"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "f7bc733173c6b03e1c488b68b78df940"
server_key_iv: "07ee4fccf7d85dd58eb0bf8d3cd358c0"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.364857 10757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:23:28.367560 10771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.368862 10757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:28.369002 10757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "018b703e741b4353b4695d6d039d68a9"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "f7bc733173c6b03e1c488b68b78df940"
server_key_iv: "07ee4fccf7d85dd58eb0bf8d3cd358c0"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.369113 10757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:28.395879 10757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:28.396636 10757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:28.396855 10757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:28.397498 10757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:28.398527 10757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:28.398574 10757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.398645 10757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:28.398675 10757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.407582 10757 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:37597
I20260923 17:23:28.407605 10885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:37597 every 8 connection(s)
I20260923 17:23:28.408504 10757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:28.417150 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 10757
I20260923 17:23:28.417322 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:23:28.417637 24410 external_mini_cluster.cc:1428] Setting key dd96591b59ec9a143662a1429da7d36a
I20260923 17:23:28.418102 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.410306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:54953 (local address 127.23.214.190:44735)
0923 17:23:28.410475 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.410479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.411480 (+ 1001us) server_negotiation.cc:407] Connection header received
0923 17:23:28.412979 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.412982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.413050 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.413165 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:28.414164 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.414722 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.415938 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.416170 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.416292 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.417537 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:28.417540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:28.417543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:28.417565 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:28.417585 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:28.417593 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:28.417673 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.417856 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.417951 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:23:28.419143 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.410663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44735 (local address 127.23.214.129:54953)
0923 17:23:28.411325 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.411366 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.412747 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.413346 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.413353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.413438 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:28.414013 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.414030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.414864 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.414868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.415785 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.415794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.416343 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.416347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.416530 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.417220 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:28.417222 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:28.417272 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:28.417378 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:28.417677 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:28.417689 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:28.417698 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.417855 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.418182 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260923 17:23:28.419837 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44735
--builtin_ntp_servers=127.23.214.148:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:23:28.420467 10886 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44735
I20260923 17:23:28.420706 10886 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:28.421281 10886 heartbeater.cc:511] Master 127.23.214.190:44735 requested a full tablet report, sending...
I20260923 17:23:28.422868 10696 ts_manager.cc:194] Registered new tserver with Master: 018b703e741b4353b4695d6d039d68a9 (127.23.214.129:37597)
I20260923 17:23:28.424108 10696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.129:54953
W20260923 17:23:28.535709 10889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:28.536008 10889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:28.536079 10889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:28.540202 10889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:28.540313 10889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:28.540402 10889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:23:28.545397 10889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:28.546594 10889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:28.547960 10889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:28.557626 10897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:28.557626 10894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:28.557657 10889 server_base.cc:1034] running on GCE node
W20260923 17:23:28.557906 10895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:28.558338 10889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:28.558995 10889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:28.560185 10889 hybrid_clock.cc:648] HybridClock initialized: now 1790184208560167 us; error 30 us; skew 500 ppm
I20260923 17:23:28.562177 10889 webserver.cc:460] Webserver started at http://127.23.214.130:37975/ using document root <none> and password file <none>
I20260923 17:23:28.562770 10889 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:28.562863 10889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:28.563103 10889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:28.565177 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "f75adfdee3d84519b79e207e01f224fb"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "574d7346fa592fd5a008a76920048452"
server_key_iv: "c7e95a526c5746b46b9a7c6e6d9bb469"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.565745 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "f75adfdee3d84519b79e207e01f224fb"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "574d7346fa592fd5a008a76920048452"
server_key_iv: "c7e95a526c5746b46b9a7c6e6d9bb469"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.569550 10889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:28.572206 10903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.573549 10889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:23:28.573668 10889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "f75adfdee3d84519b79e207e01f224fb"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "574d7346fa592fd5a008a76920048452"
server_key_iv: "c7e95a526c5746b46b9a7c6e6d9bb469"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.573782 10889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:28.586514 10889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:28.587219 10889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:28.587422 10889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:28.588027 10889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:28.588984 10889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:28.589032 10889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.589072 10889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:28.589123 10889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.598342 10889 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:34469
I20260923 17:23:28.598407 11017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:34469 every 8 connection(s)
I20260923 17:23:28.599323 10889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:28.606765 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 10889
I20260923 17:23:28.606905 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:23:28.607228 24410 external_mini_cluster.cc:1428] Setting key 7d67596cd07305ff8a228d430a2eae78
I20260923 17:23:28.609300 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44735
--builtin_ntp_servers=127.23.214.148:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:23:28.609447 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.601344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:32805 (local address 127.23.214.190:44735)
0923 17:23:28.601477 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.601481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.602612 (+ 1131us) server_negotiation.cc:407] Connection header received
0923 17:23:28.604058 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.604061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.604111 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.604195 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:28.605116 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.605662 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.607110 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.607422 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.607560 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.608943 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:28.608946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:28.608949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:28.608960 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:28.608980 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:28.608988 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:28.609068 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.609235 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.609304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:23:28.610476 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.601763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44735 (local address 127.23.214.130:32805)
0923 17:23:28.602456 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.602497 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.603825 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.604362 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.604369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.604457 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:28.604972 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.604985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.605813 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.605815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.606959 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.606971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.607621 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.607627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.607884 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.608654 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:28.608659 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:28.608714 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:28.608783 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:28.609072 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:28.609083 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:28.609090 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.609223 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.609535 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":163,"threads_started":1}
I20260923 17:23:28.611727 11018 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44735
I20260923 17:23:28.611984 11018 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:28.612519 11018 heartbeater.cc:511] Master 127.23.214.190:44735 requested a full tablet report, sending...
I20260923 17:23:28.613715 10696 ts_manager.cc:194] Registered new tserver with Master: f75adfdee3d84519b79e207e01f224fb (127.23.214.130:34469)
I20260923 17:23:28.614393 10696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.130:32805
W20260923 17:23:28.721416 11021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:28.721693 11021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:28.721733 11021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:28.725796 11021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:28.725852 11021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:28.725981 11021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:23:28.730759 11021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:44735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:28.731844 11021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:28.733208 11021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:28.742667 11029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:28.742745 11026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:28.742679 11027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:28.743086 11021 server_base.cc:1034] running on GCE node
I20260923 17:23:28.743474 11021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:28.744029 11021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:28.745185 11021 hybrid_clock.cc:648] HybridClock initialized: now 1790184208745172 us; error 23 us; skew 500 ppm
I20260923 17:23:28.746994 11021 webserver.cc:460] Webserver started at http://127.23.214.131:37671/ using document root <none> and password file <none>
I20260923 17:23:28.747618 11021 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:28.747684 11021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:28.747896 11021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:28.749965 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "54fb92b7d1f24dbd9f6471633ad45eb7"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "e69607807d33f3f7a32ce6716ecdd265"
server_key_iv: "d208e938c73f373143de610f5880dea0"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.750541 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "54fb92b7d1f24dbd9f6471633ad45eb7"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "e69607807d33f3f7a32ce6716ecdd265"
server_key_iv: "d208e938c73f373143de610f5880dea0"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.754293 11021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:23:28.756565 11035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.757802 11021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:28.757952 11021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "54fb92b7d1f24dbd9f6471633ad45eb7"
format_stamp: "Formatted at 2026-09-23 17:23:28 on dist-test-slave-9q7f"
server_key: "e69607807d33f3f7a32ce6716ecdd265"
server_key_iv: "d208e938c73f373143de610f5880dea0"
server_key_version: "encryptionkey@0"
I20260923 17:23:28.758057 11021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:28.786896 11021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:28.787551 11021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:28.787717 11021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:28.788246 11021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:28.789149 11021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:28.789196 11021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.789235 11021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:28.789251 11021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:28.798247 11021 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:42125
I20260923 17:23:28.798264 11149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:42125 every 8 connection(s)
I20260923 17:23:28.799149 11021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:28.806459 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11021
I20260923 17:23:28.806567 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:23:28.806892 24410 external_mini_cluster.cc:1428] Setting key ccbc2daa5719d9dd8906cc5b44e7f84f
I20260923 17:23:28.808065 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.800932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:60781 (local address 127.23.214.190:44735)
0923 17:23:28.801068 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.801072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.802135 (+ 1063us) server_negotiation.cc:407] Connection header received
0923 17:23:28.803465 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.803467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.803512 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.803585 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:28.804426 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.804956 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.806039 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.806263 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.806336 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.807568 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:28.807572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:28.807574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:28.807586 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:28.807613 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:28.807621 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:28.807701 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.807869 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.807928 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:23:28.808833 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.801327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:44735 (local address 127.23.214.131:60781)
0923 17:23:28.801994 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.802034 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.803281 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.803764 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.803770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.803849 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:28.804307 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.804319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.805058 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.805061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.805944 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.805952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.806406 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.806411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.806614 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.807307 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:28.807308 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:28.807346 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:28.807404 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:28.807713 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:28.807723 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:28.807736 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.807852 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.808104 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260923 17:23:28.809926 11150 heartbeater.cc:347] Connected to a master server at 127.23.214.190:44735
I20260923 17:23:28.810164 11150 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:28.810667 11150 heartbeater.cc:511] Master 127.23.214.190:44735 requested a full tablet report, sending...
I20260923 17:23:28.811599 10696 ts_manager.cc:194] Registered new tserver with Master: 54fb92b7d1f24dbd9f6471633ad45eb7 (127.23.214.131:42125)
I20260923 17:23:28.812142 10696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.131:60781
I20260923 17:23:28.820842 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:23:28.827062 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.822875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.23.214.190:44735)
0923 17:23:28.823036 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.823041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.823167 (+ 126us) server_negotiation.cc:407] Connection header received
0923 17:23:28.823319 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.823322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.823369 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.823453 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:28.823815 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.824332 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.825254 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.825490 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.825592 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.826567 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:28.826570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:28.826572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:28.826584 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:28.826603 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:28.826611 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:28.826753 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.826879 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.826927 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:23:28.830116 10696 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:23:28.833169 10696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:23:28.847103 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.841346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.23.214.130:34469)
0923 17:23:28.841658 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.841682 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.841706 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:23:28.841816 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.841820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.842266 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.842442 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:28.842947 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.843917 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.845532 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.846353 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.846471 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.846646 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.846838 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.846934 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":133,"threads_started":1}
I20260923 17:23:28.847342 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.841192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:34469 (local address 127.0.0.1:36920)
0923 17:23:28.841509 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.841554 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.841671 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.842633 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.842639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.842666 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:28.842805 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.842815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.844026 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.844029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.845387 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.845397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.846475 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.846479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.846651 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.846694 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.846807 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.846926 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":46,"thread_start_us":76,"threads_started":1}
I20260923 17:23:28.847779 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.842281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37597 (local address 127.0.0.1:37490)
0923 17:23:28.843329 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.843341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.843424 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.843997 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.844000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.844021 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:28.844137 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.844142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.845096 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.845101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.846155 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.846164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.847405 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.847409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.847503 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.847536 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.847605 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.847675 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":29,"thread_start_us":88,"threads_started":1}
I20260923 17:23:28.848239 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.842467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.23.214.129:37597)
0923 17:23:28.842928 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.842940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.843444 (+ 504us) server_negotiation.cc:407] Connection header received
0923 17:23:28.843540 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.843545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.843725 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.843858 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:28.844249 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.844972 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.846272 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.847034 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.847175 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.847340 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.847953 (+ 613us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.848079 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":142,"threads_started":1}
I20260923 17:23:28.849069 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.842683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.23.214.131:42125)
0923 17:23:28.843152 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.843163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.843189 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:23:28.843272 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.843277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.843494 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.843652 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:28.843842 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.844974 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.846731 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.847496 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.847622 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.847799 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.848094 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.848189 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":111,"threads_started":1}
I20260923 17:23:28.849973 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.841913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:42125 (local address 127.0.0.1:37972)
0923 17:23:28.843059 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.843074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.843160 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.843626 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.843629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.843645 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:28.843733 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.843739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.845347 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.845351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.846631 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.846641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.847862 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.847865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.847951 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.847973 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.848051 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.848112 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":68,"thread_start_us":104,"threads_started":1}
I20260923 17:23:28.850582 10951 tablet_service.cc:1467] Processing CreateTablet for tablet 389be5db1b4f4729a803e7c045d103f7 (DEFAULT_TABLE table=test-table [id=941fe35163f44b45845ecf5a518d6b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:28.850950 10819 tablet_service.cc:1467] Processing CreateTablet for tablet 389be5db1b4f4729a803e7c045d103f7 (DEFAULT_TABLE table=test-table [id=941fe35163f44b45845ecf5a518d6b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:28.851728 10951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389be5db1b4f4729a803e7c045d103f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:28.851895 10819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389be5db1b4f4729a803e7c045d103f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:28.852326 11083 tablet_service.cc:1467] Processing CreateTablet for tablet 389be5db1b4f4729a803e7c045d103f7 (DEFAULT_TABLE table=test-table [id=941fe35163f44b45845ecf5a518d6b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:28.853116 11083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389be5db1b4f4729a803e7c045d103f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:28.858560 11167 tablet_bootstrap.cc:492] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Bootstrap starting.
I20260923 17:23:28.859630 11168 tablet_bootstrap.cc:492] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: Bootstrap starting.
I20260923 17:23:28.859630 11169 tablet_bootstrap.cc:492] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Bootstrap starting.
I20260923 17:23:28.860859 11167 tablet_bootstrap.cc:654] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:28.861724 11169 tablet_bootstrap.cc:654] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:28.861924 11167 log.cc:824] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:28.862035 11168 tablet_bootstrap.cc:654] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:28.862603 11169 log.cc:824] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:28.862854 11168 log.cc:824] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:28.864053 11167 tablet_bootstrap.cc:492] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: No bootstrap required, opened a new log
I20260923 17:23:28.864269 11167 ts_tablet_manager.cc:1397] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:23:28.864547 11169 tablet_bootstrap.cc:492] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: No bootstrap required, opened a new log
I20260923 17:23:28.864595 11168 tablet_bootstrap.cc:492] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: No bootstrap required, opened a new log
I20260923 17:23:28.864729 11169 ts_tablet_manager.cc:1397] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:23:28.864728 11168 ts_tablet_manager.cc:1397] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:23:28.868273 11167 raft_consensus.cc:348] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.868501 11167 raft_consensus.cc:374] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:28.868552 11167 raft_consensus.cc:729] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fb92b7d1f24dbd9f6471633ad45eb7, State: Initialized, Role: FOLLOWER
I20260923 17:23:28.868935 11169 raft_consensus.cc:348] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.869059 11167 consensus_queue.cc:260] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.869196 11169 raft_consensus.cc:374] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:28.869257 11169 raft_consensus.cc:729] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75adfdee3d84519b79e207e01f224fb, State: Initialized, Role: FOLLOWER
I20260923 17:23:28.869119 11168 raft_consensus.cc:348] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.869330 11168 raft_consensus.cc:374] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:28.869382 11168 raft_consensus.cc:729] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 018b703e741b4353b4695d6d039d68a9, State: Initialized, Role: FOLLOWER
I20260923 17:23:28.869688 11169 consensus_queue.cc:260] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.869815 11168 consensus_queue.cc:260] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.870038 11167 ts_tablet_manager.cc:1428] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:23:28.870416 11150 heartbeater.cc:503] Master 127.23.214.190:44735 was elected leader, sending a full tablet report...
I20260923 17:23:28.870566 10886 heartbeater.cc:503] Master 127.23.214.190:44735 was elected leader, sending a full tablet report...
I20260923 17:23:28.870677 11169 ts_tablet_manager.cc:1428] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 17:23:28.870683 11168 ts_tablet_manager.cc:1428] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:23:28.870982 11018 heartbeater.cc:503] Master 127.23.214.190:44735 was elected leader, sending a full tablet report...
W20260923 17:23:28.909665 10887 tablet.cc:2369] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:28.946237 11173 raft_consensus.cc:482] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:28.946511 11173 raft_consensus.cc:504] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.947854 11173 leader_election.cc:288] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f75adfdee3d84519b79e207e01f224fb (127.23.214.130:34469), 018b703e741b4353b4695d6d039d68a9 (127.23.214.129:37597)
I20260923 17:23:28.952921 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.948468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:52203 (local address 127.23.214.129:37597)
0923 17:23:28.948614 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.948619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.948821 (+ 202us) server_negotiation.cc:407] Connection header received
0923 17:23:28.948929 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.948931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.948969 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.949048 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:28.949312 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.950043 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.951686 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.952298 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.952416 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.952465 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.952715 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.952788 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:23:28.952921 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.948156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:34151 (local address 127.23.214.130:34469)
0923 17:23:28.948302 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.948308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.948325 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:23:28.948591 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.948594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.948653 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.948720 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:28.949090 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.949940 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.951686 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.952286 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.952416 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.952465 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.952715 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.952797 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:23:28.952924 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.948040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:34469 (local address 127.23.214.131:34151)
0923 17:23:28.948173 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.948191 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.948315 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.948760 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.948765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.948786 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:28.948955 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.948963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.950047 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.950051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.951551 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.951563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.952420 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.952423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.952566 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.952613 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.952688 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.952772 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":153}
I20260923 17:23:28.952934 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.948327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37597 (local address 127.23.214.131:52203)
0923 17:23:28.948682 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:28.948720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:28.948847 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:28.949064 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:28.949069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:28.949087 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:28.949191 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.949195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.950132 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.950134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.951551 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:28.951563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.952417 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.952421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:28.952563 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.952610 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 17:23:28.952688 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:23:28.952772 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":144,"thread_start_us":126,"threads_started":1}
I20260923 17:23:28.953724 10971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389be5db1b4f4729a803e7c045d103f7" candidate_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75adfdee3d84519b79e207e01f224fb" is_pre_election: true
I20260923 17:23:28.953733 10839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389be5db1b4f4729a803e7c045d103f7" candidate_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "018b703e741b4353b4695d6d039d68a9" is_pre_election: true
I20260923 17:23:28.954123 10839 raft_consensus.cc:2454] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54fb92b7d1f24dbd9f6471633ad45eb7 in term 0.
I20260923 17:23:28.954138 10971 raft_consensus.cc:2454] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54fb92b7d1f24dbd9f6471633ad45eb7 in term 0.
I20260923 17:23:28.954612 11036 leader_election.cc:302] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54fb92b7d1f24dbd9f6471633ad45eb7, f75adfdee3d84519b79e207e01f224fb; no voters:
I20260923 17:23:28.954943 11173 raft_consensus.cc:2781] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:23:28.955029 11173 raft_consensus.cc:482] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:23:28.955078 11173 raft_consensus.cc:3037] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:28.956398 11173 raft_consensus.cc:504] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.956781 11173 leader_election.cc:288] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [CANDIDATE]: Term 1 election: Requested vote from peers f75adfdee3d84519b79e207e01f224fb (127.23.214.130:34469), 018b703e741b4353b4695d6d039d68a9 (127.23.214.129:37597)
I20260923 17:23:28.957120 10839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389be5db1b4f4729a803e7c045d103f7" candidate_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "018b703e741b4353b4695d6d039d68a9"
I20260923 17:23:28.957125 10971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389be5db1b4f4729a803e7c045d103f7" candidate_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75adfdee3d84519b79e207e01f224fb"
I20260923 17:23:28.957239 10971 raft_consensus.cc:3037] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:28.957250 10839 raft_consensus.cc:3037] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:28.958324 10971 raft_consensus.cc:2454] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54fb92b7d1f24dbd9f6471633ad45eb7 in term 1.
I20260923 17:23:28.958453 10839 raft_consensus.cc:2454] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54fb92b7d1f24dbd9f6471633ad45eb7 in term 1.
I20260923 17:23:28.958607 11036 leader_election.cc:302] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54fb92b7d1f24dbd9f6471633ad45eb7, f75adfdee3d84519b79e207e01f224fb; no voters:
I20260923 17:23:28.958797 11173 raft_consensus.cc:2781] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:28.959029 11173 raft_consensus.cc:686] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 1 LEADER]: Becoming Leader. State: Replica: 54fb92b7d1f24dbd9f6471633ad45eb7, State: Running, Role: LEADER
I20260923 17:23:28.959362 11173 consensus_queue.cc:237] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } }
I20260923 17:23:28.962743 10695 catalog_manager.cc:5495] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54fb92b7d1f24dbd9f6471633ad45eb7 (127.23.214.131). New cstate: current_term: 1 leader_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54fb92b7d1f24dbd9f6471633ad45eb7" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 42125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 } health_report { overall_health: UNKNOWN } } }
I20260923 17:23:28.979976 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:28.976532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.23.214.131:42125)
0923 17:23:28.976655 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:28.976659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:28.976672 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:23:28.976741 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:28.976743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:28.976809 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:28.976893 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:23:28.977097 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.977667 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.978609 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:28.978834 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:28.978916 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:28.979200 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:23:28.979598 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:23:28.979707 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:28.979830 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 17:23:28.979864 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:23:28.986011 10971 raft_consensus.cc:1261] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 1 FOLLOWER]: Refusing update from remote peer 54fb92b7d1f24dbd9f6471633ad45eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:28.986011 10839 raft_consensus.cc:1261] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 1 FOLLOWER]: Refusing update from remote peer 54fb92b7d1f24dbd9f6471633ad45eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:28.986773 11173 consensus_queue.cc:1035] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "018b703e741b4353b4695d6d039d68a9" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:28.986970 11177 consensus_queue.cc:1035] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75adfdee3d84519b79e207e01f224fb" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:29.003865 10693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:29.004058 10693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:29.006457 10693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd2b57e183d949ec95ac06685881c674: Sending DeleteTablet for 3 replicas of tablet 389be5db1b4f4729a803e7c045d103f7
I20260923 17:23:29.007262 10951 tablet_service.cc:1514] Processing DeleteTablet for tablet 389be5db1b4f4729a803e7c045d103f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:28 UTC) from {username='slave'} at 127.0.0.1:36920
I20260923 17:23:29.007457 10819 tablet_service.cc:1514] Processing DeleteTablet for tablet 389be5db1b4f4729a803e7c045d103f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:28 UTC) from {username='slave'} at 127.0.0.1:37490
I20260923 17:23:29.007468 11083 tablet_service.cc:1514] Processing DeleteTablet for tablet 389be5db1b4f4729a803e7c045d103f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:28 UTC) from {username='slave'} at 127.0.0.1:37972
I20260923 17:23:29.007954 11192 tablet_replica.cc:330] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: stopping tablet replica
I20260923 17:23:29.007961 11191 tablet_replica.cc:330] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9: stopping tablet replica
I20260923 17:23:29.008288 11191 raft_consensus.cc:2229] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:29.008381 11192 raft_consensus.cc:2229] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:23:29.008610 11191 raft_consensus.cc:2258] T 389be5db1b4f4729a803e7c045d103f7 P 018b703e741b4353b4695d6d039d68a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:29.008693 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 10757
I20260923 17:23:29.008720 11192 raft_consensus.cc:2258] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:29.011468 11190 tablet_replica.cc:330] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: stopping tablet replica
I20260923 17:23:29.011770 11190 raft_consensus.cc:2229] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:29.012075 11190 raft_consensus.cc:2258] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:29.012197 11192 ts_tablet_manager.cc:1905] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:29.013612 11190 ts_tablet_manager.cc:1905] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:29.015702 11192 ts_tablet_manager.cc:1918] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:29.015825 11192 log.cc:1196] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/389be5db1b4f4729a803e7c045d103f7
W20260923 17:23:29.015998 10680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:37597 (error 108)
I20260923 17:23:29.016162 11192 ts_tablet_manager.cc:1939] T 389be5db1b4f4729a803e7c045d103f7 P 54fb92b7d1f24dbd9f6471633ad45eb7: Deleting consensus metadata
I20260923 17:23:29.016577 11190 ts_tablet_manager.cc:1918] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:29.016731 11190 log.cc:1196] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/389be5db1b4f4729a803e7c045d103f7
I20260923 17:23:29.016814 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 10889
I20260923 17:23:29.017113 11190 ts_tablet_manager.cc:1939] T 389be5db1b4f4729a803e7c045d103f7 P f75adfdee3d84519b79e207e01f224fb: Deleting consensus metadata
I20260923 17:23:29.018838 10680 catalog_manager.cc:4841] TS 54fb92b7d1f24dbd9f6471633ad45eb7 (127.23.214.131:42125): tablet 389be5db1b4f4729a803e7c045d103f7 (table test-table [id=941fe35163f44b45845ecf5a518d6b28]) successfully deleted
I20260923 17:23:29.018966 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:29.018659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:37597 (local address 127.0.0.1:37504)
0923 17:23:29.018808 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:29.018881 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:23:29.019196 10680 catalog_manager.cc:4584] TS 018b703e741b4353b4695d6d039d68a9 (127.23.214.129:37597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 389be5db1b4f4729a803e7c045d103f7: Network error: Client connection negotiation failed: client connection to 127.23.214.129:37597: connect: Connection refused (error 111)
I20260923 17:23:29.023630 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11021
I20260923 17:23:29.024577 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:29.023758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:34469 (local address 127.0.0.1:36928)
0923 17:23:29.024412 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:29.024475 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.130:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563}
I20260923 17:23:29.029718 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 10665
2026-09-23T17:23:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:29 dist-test-slave-9q7f krb5kdc[11196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:29 dist-test-slave-9q7f krb5kdc[11196](info): set up 2 sockets
Sep 23 17:23:29 dist-test-slave-9q7f krb5kdc[11196](info): commencing operation
krb5kdc: starting...
W20260923 17:23:30.075980 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:30 dist-test-slave-9q7f krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:30.202752 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40671
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:40671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:30.316908 11209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:30.317224 11209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:30.317309 11209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:30.321461 11209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:30.321564 11209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:30.321604 11209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:30.321650 11209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:30.321686 11209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:30.327013 11209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:40671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:40671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:30.328168 11209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:30.329212 11209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:30.338112 11217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:30.338112 11215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:30.338160 11214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:30.338506 11209 server_base.cc:1034] running on GCE node
I20260923 17:23:30.339062 11209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:30.340185 11209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:30.341351 11209 hybrid_clock.cc:648] HybridClock initialized: now 1790184210341319 us; error 46 us; skew 500 ppm
Sep 23 17:23:30 dist-test-slave-9q7f krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:30.350740 11209 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:30.351704 11209 webserver.cc:460] Webserver started at http://127.23.214.190:40939/ using document root <none> and password file <none>
I20260923 17:23:30.352349 11209 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:30.352443 11209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:30.352679 11209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:30.354846 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "b477c80581834e44a258f82337c991a8"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "c724810dfb9d0d306bbaf6a37a2017b3"
server_key_iv: "5fccf0d24f8010b3a846836625cf992d"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.355444 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "b477c80581834e44a258f82337c991a8"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "c724810dfb9d0d306bbaf6a37a2017b3"
server_key_iv: "5fccf0d24f8010b3a846836625cf992d"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.359334 11209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:23:30.361771 11224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:30.363124 11209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:30.363283 11209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "b477c80581834e44a258f82337c991a8"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "c724810dfb9d0d306bbaf6a37a2017b3"
server_key_iv: "5fccf0d24f8010b3a846836625cf992d"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.363420 11209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:30.374382 11209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:30.377177 11209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:30.377391 11209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:30.384668 11209 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:40671
I20260923 17:23:30.384694 11278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:40671 every 8 connection(s)
I20260923 17:23:30.385728 11209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:30.388520 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:30.389518 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11209
I20260923 17:23:30.389695 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:30.390069 24410 external_mini_cluster.cc:1428] Setting key ed0eab27d1b7271a4190dc89500a3d99
I20260923 17:23:30.395385 11279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8: Bootstrap starting.
I20260923 17:23:30.398116 11279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:30.398904 11279 log.cc:824] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:23:30 dist-test-slave-9q7f krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:30.401278 11279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8: No bootstrap required, opened a new log
I20260923 17:23:30.405541 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:30.391538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.23.214.190:40671)
0923 17:23:30.391968 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:30.391988 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:30.392031 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:23:30.392856 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:30.392878 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:30.393294 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:30.393659 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:30.394426 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.395861 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.396730 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.397420 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.397577 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:30.400320 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:30.400326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:30.400345 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:30.400383 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:30.402383 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:30.402810 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:30.402816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:30.402823 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:30.402960 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:30.403214 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:30.403217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:30.403221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:30.403804 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:30.404043 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:30.404229 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:23:30.404546 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
I20260923 17:23:30.406075 11279 raft_consensus.cc:348] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477c80581834e44a258f82337c991a8" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40671 } }
I20260923 17:23:30.406297 11279 raft_consensus.cc:374] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:30.406354 11279 raft_consensus.cc:729] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b477c80581834e44a258f82337c991a8, State: Initialized, Role: FOLLOWER
I20260923 17:23:30.406800 11279 consensus_queue.cc:260] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477c80581834e44a258f82337c991a8" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40671 } }
I20260923 17:23:30.406967 11279 raft_consensus.cc:388] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:30.407054 11279 raft_consensus.cc:482] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:30.407191 11279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:30.408144 11279 raft_consensus.cc:504] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477c80581834e44a258f82337c991a8" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40671 } }
I20260923 17:23:30.408499 11279 leader_election.cc:302] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b477c80581834e44a258f82337c991a8; no voters:
I20260923 17:23:30.408830 11279 leader_election.cc:288] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:30.409065 11284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:30.409392 11284 raft_consensus.cc:686] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [term 1 LEADER]: Becoming Leader. State: Replica: b477c80581834e44a258f82337c991a8, State: Running, Role: LEADER
I20260923 17:23:30.409726 11284 consensus_queue.cc:237] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477c80581834e44a258f82337c991a8" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40671 } }
I20260923 17:23:30.410195 11279 sys_catalog.cc:564] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:30.411653 11286 sys_catalog.cc:455] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b477c80581834e44a258f82337c991a8. Latest consensus state: current_term: 1 leader_uuid: "b477c80581834e44a258f82337c991a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477c80581834e44a258f82337c991a8" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40671 } } }
I20260923 17:23:30.411788 11286 sys_catalog.cc:458] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:30.412112 11285 sys_catalog.cc:455] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b477c80581834e44a258f82337c991a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477c80581834e44a258f82337c991a8" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 40671 } } }
I20260923 17:23:30.412205 11285 sys_catalog.cc:458] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:30.412539 11295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:30.415648 11295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:30.421697 11295 catalog_manager.cc:1347] Generated new cluster ID: 1fcbc07b247d49cd8c328a01ca27788b
I20260923 17:23:30.421777 11295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:30.430730 11295 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:30.431712 11295 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:30.436359 11295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8: Generated new TSK 0
I20260923 17:23:30.437031 11295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.214.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
Entry for principal kudu/127.23.214.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab.
I20260923 17:23:30.472749 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40671
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:30.584599 11306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:30.584879 11306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:30.584923 11306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:30.588920 11306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:30.588979 11306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:30.589080 11306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:23:30.594336 11306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.129.keytab
--principal=kudu/127.23.214.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:30.595378 11306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:30.596426 11306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:30.605851 11312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:30.605939 11306 server_base.cc:1034] running on GCE node
W20260923 17:23:30.605815 11311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:30.605837 11314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:30.606510 11306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:30.607129 11306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:30.608295 11306 hybrid_clock.cc:648] HybridClock initialized: now 1790184210608276 us; error 31 us; skew 500 ppm
Sep 23 17:23:30 dist-test-slave-9q7f krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:30.611194 11306 init.cc:377] Logged in from keytab as kudu/127.23.214.129@KRBTEST.COM (short username kudu)
I20260923 17:23:30.612206 11306 webserver.cc:460] Webserver started at http://127.23.214.129:36101/ using document root <none> and password file <none>
I20260923 17:23:30.612778 11306 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:30.612867 11306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:30.613101 11306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:30.615161 11306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "af9a3ee323a58be3172fb39cf5d93ef8"
server_key_iv: "de1f95c544cba80a405198fc6404ee84"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.615746 11306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "af9a3ee323a58be3172fb39cf5d93ef8"
server_key_iv: "de1f95c544cba80a405198fc6404ee84"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.619369 11306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:23:30.621722 11321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:30.623018 11306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:30.623153 11306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "af9a3ee323a58be3172fb39cf5d93ef8"
server_key_iv: "de1f95c544cba80a405198fc6404ee84"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.623271 11306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:30.633040 11306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:30.635845 11306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:30.636039 11306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:30.636606 11306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:30.637411 11306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:30.637480 11306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:30.637550 11306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:30.637604 11306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:30.646384 11306 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:40811
I20260923 17:23:30.646405 11435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:40811 every 8 connection(s)
I20260923 17:23:30.647261 11306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:30.650020 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11306
I20260923 17:23:30.650159 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:23:30.650542 24410 external_mini_cluster.cc:1428] Setting key 85b014c9098fa1c93d0599b6dff314d2
Sep 23 17:23:30 dist-test-slave-9q7f krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.129@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.130@KRBTEST.COM; defaulting to no policy
I20260923 17:23:30.662719 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:30.649082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:53577 (local address 127.23.214.190:40671)
0923 17:23:30.649223 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:30.649234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:30.650158 (+ 924us) server_negotiation.cc:407] Connection header received
0923 17:23:30.651835 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:30.651839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:30.651888 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:30.651982 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:30.653556 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.654223 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.655718 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.656145 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.656275 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:30.659087 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:30.659091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:30.659094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:30.659127 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:30.660631 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:30.661142 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:30.661147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:30.661149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:30.661234 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:30.661576 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:30.661581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:30.661582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:30.661776 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:30.661907 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:30.662415 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 17:23:30.662539 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.130@KRBTEST.COM" created.
I20260923 17:23:30.663744 11438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:30.649453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40671 (local address 127.23.214.129:53577)
0923 17:23:30.649996 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:30.650040 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:30.651595 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:30.652172 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:30.652181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:30.652704 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:30.653400 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:30.653422 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.654368 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.654375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:30.655588 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:30.655598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.656301 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.656304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:30.656525 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:30.657191 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:30.657213 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:30.658915 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:30.660756 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:30.660762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:30.660778 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:30.661035 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:30.661328 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:30.661332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:30.661335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:30.661478 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:30.661901 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:30.661908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:30.662171 (+ 263us) client_negotiation.cc:769] Sending connection context
0923 17:23:30.662375 (+ 204us) client_negotiation.cc:240] Negotiation successful
0923 17:23:30.662710 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260923 17:23:30.665053 11436 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40671
I20260923 17:23:30.665333 11436 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:30.665961 11436 heartbeater.cc:511] Master 127.23.214.190:40671 requested a full tablet report, sending...
I20260923 17:23:30.667546 11242 ts_manager.cc:194] Registered new tserver with Master: 1ba4b3fa7d6e4cdc9318dfb56e25d66e (127.23.214.129:40811)
I20260923 17:23:30.668741 11242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.129@KRBTEST.COM'} at 127.23.214.129:53577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
Entry for principal kudu/127.23.214.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab.
I20260923 17:23:30.680986 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40671
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:30.791286 11441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:30.791555 11441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:30.791599 11441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:30.795646 11441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:30.795707 11441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:30.795766 11441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:23:30.801183 11441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.130.keytab
--principal=kudu/127.23.214.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:30.802474 11441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:30.803507 11441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:30.813153 11446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:30.813232 11449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:30.813308 11441 server_base.cc:1034] running on GCE node
W20260923 17:23:30.813218 11447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:30.813957 11441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:30.814606 11441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:30.815790 11441 hybrid_clock.cc:648] HybridClock initialized: now 1790184210815755 us; error 46 us; skew 500 ppm
Sep 23 17:23:30 dist-test-slave-9q7f krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:30.818441 11441 init.cc:377] Logged in from keytab as kudu/127.23.214.130@KRBTEST.COM (short username kudu)
I20260923 17:23:30.819305 11441 webserver.cc:460] Webserver started at http://127.23.214.130:37247/ using document root <none> and password file <none>
I20260923 17:23:30.819857 11441 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:30.819947 11441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:30.820173 11441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:30.822304 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "f00b9a3ce11e464a891413de36a042e2"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "be1f3b0c56b7e1fa4c2b422b5eef08d9"
server_key_iv: "cb300ab57d241629ae168996d1d6e76a"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.822939 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "f00b9a3ce11e464a891413de36a042e2"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "be1f3b0c56b7e1fa4c2b422b5eef08d9"
server_key_iv: "cb300ab57d241629ae168996d1d6e76a"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.826766 11441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:23:30.829077 11456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:30.830361 11441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:30.830489 11441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "f00b9a3ce11e464a891413de36a042e2"
format_stamp: "Formatted at 2026-09-23 17:23:30 on dist-test-slave-9q7f"
server_key: "be1f3b0c56b7e1fa4c2b422b5eef08d9"
server_key_iv: "cb300ab57d241629ae168996d1d6e76a"
server_key_version: "encryptionkey@0"
I20260923 17:23:30.830595 11441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:30.854700 11441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:30.857649 11441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:30.857862 11441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:30.858548 11441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:30.859380 11441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:30.859449 11441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:30.859519 11441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:30.859570 11441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:30.868878 11441 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:34903
I20260923 17:23:30.868899 11570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:34903 every 8 connection(s)
I20260923 17:23:30.869792 11441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:30.877846 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11441
I20260923 17:23:30.878053 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:23:30.878432 24410 external_mini_cluster.cc:1428] Setting key 943511267c9dcbd06601680174c522f3
Sep 23 17:23:30 dist-test-slave-9q7f krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.130@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:30.883713 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:30.871608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:56959 (local address 127.23.214.190:40671)
0923 17:23:30.871769 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:30.871772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:30.872575 (+ 803us) server_negotiation.cc:407] Connection header received
0923 17:23:30.873932 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:30.873935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:30.873992 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:30.874097 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:30.875263 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.875793 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.876917 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.877138 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.877255 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:30.880156 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:30.880159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:30.880161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:30.880183 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:30.881640 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:30.882225 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:30.882230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:30.882231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:30.882321 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:30.882663 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:30.882668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:30.882670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:30.882846 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:30.882952 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:30.883436 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 17:23:30.883550 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:23:30.884497 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:30.872002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40671 (local address 127.23.214.130:56959)
0923 17:23:30.872436 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:30.872477 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:30.873687 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:30.874268 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:30.874275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:30.874624 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:30.875106 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:30.875119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.875920 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.875928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:30.876791 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:30.876799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:30.877293 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:30.877297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:30.877460 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:30.878337 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:30.878360 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:30.879990 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:30.881762 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:30.881768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:30.881781 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:30.882121 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:30.882416 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:30.882420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:30.882422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:30.882565 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:30.882949 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:30.882956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:30.883199 (+ 243us) client_negotiation.cc:769] Sending connection context
0923 17:23:30.883379 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 17:23:30.883688 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260923 17:23:30.885667 11571 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40671
I20260923 17:23:30.885946 11571 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:30.886538 11571 heartbeater.cc:511] Master 127.23.214.190:40671 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.214.131@KRBTEST.COM; defaulting to no policy
I20260923 17:23:30.887676 11242 ts_manager.cc:194] Registered new tserver with Master: f00b9a3ce11e464a891413de36a042e2 (127.23.214.130:34903)
I20260923 17:23:30.888250 11242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.130@KRBTEST.COM'} at 127.23.214.130:56959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
Entry for principal kudu/127.23.214.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab.
I20260923 17:23:30.905046 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40671
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:31.019177 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:31.019445 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:31.019531 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:31.023674 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:31.023739 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:31.023846 11576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:23:31.030123 11576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.131.keytab
--principal=kudu/127.23.214.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:40671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:31.031711 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:31.033248 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:31.044065 11584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:31.044054 11582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:31.044056 11581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:31.044286 11576 server_base.cc:1034] running on GCE node
I20260923 17:23:31.044675 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:31.045341 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:31.046559 11576 hybrid_clock.cc:648] HybridClock initialized: now 1790184211046542 us; error 26 us; skew 500 ppm
Sep 23 17:23:31 dist-test-slave-9q7f krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184211, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:31.049328 11576 init.cc:377] Logged in from keytab as kudu/127.23.214.131@KRBTEST.COM (short username kudu)
I20260923 17:23:31.050343 11576 webserver.cc:460] Webserver started at http://127.23.214.131:40401/ using document root <none> and password file <none>
I20260923 17:23:31.050998 11576 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:31.051085 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:31.051353 11576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:31.053447 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "b280d60db15e4088816cb1d20c61c931"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "9e51aa3cf233260dcd87c5c590c2158a"
server_key_iv: "c11ac1e942d3d64bcdd710585e884e35"
server_key_version: "encryptionkey@0"
I20260923 17:23:31.054068 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "b280d60db15e4088816cb1d20c61c931"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "9e51aa3cf233260dcd87c5c590c2158a"
server_key_iv: "c11ac1e942d3d64bcdd710585e884e35"
server_key_version: "encryptionkey@0"
I20260923 17:23:31.057807 11576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:31.060364 11591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:31.061550 11576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:23:31.061719 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "b280d60db15e4088816cb1d20c61c931"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "9e51aa3cf233260dcd87c5c590c2158a"
server_key_iv: "c11ac1e942d3d64bcdd710585e884e35"
server_key_version: "encryptionkey@0"
I20260923 17:23:31.061836 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:31.084802 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:31.087968 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:31.088414 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:31.089164 11576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:31.090149 11576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:31.090225 11576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:31.090304 11576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:31.090379 11576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:31.100193 11576 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:36511
I20260923 17:23:31.100207 11705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:36511 every 8 connection(s)
I20260923 17:23:31.101184 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:31.101716 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11576
I20260923 17:23:31.101858 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:23:31.102278 24410 external_mini_cluster.cc:1428] Setting key b47b8016d8190c27e7adefefbae83fa0
Sep 23 17:23:31 dist-test-slave-9q7f krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184211, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.131@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:31.115893 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.103208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:56449 (local address 127.23.214.190:40671)
0923 17:23:31.103344 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.103348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.104519 (+ 1171us) server_negotiation.cc:407] Connection header received
0923 17:23:31.105945 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.105948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.106003 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.106128 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:31.107337 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.107857 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.109000 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.109233 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.109352 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.112062 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:31.112066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:31.112068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:31.112102 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:31.113510 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:31.114088 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:31.114092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:31.114094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:31.114181 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:31.114532 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:31.114536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:31.114538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:31.114729 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:31.114838 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.115528 (+ 690us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.115686 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:23:31.116878 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.103760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:40671 (local address 127.23.214.131:56449)
0923 17:23:31.104306 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:31.104378 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:31.105698 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:31.106302 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:31.106309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:31.106657 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:31.107163 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.107176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.108006 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.108009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.108876 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.108885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.109389 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.109393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.109567 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.110258 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:31.110274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:31.111896 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:31.113626 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:31.113632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:31.113648 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:31.113987 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:31.114279 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:31.114282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:31.114284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:31.114436 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:31.114838 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:31.114845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:31.115166 (+ 321us) client_negotiation.cc:769] Sending connection context
0923 17:23:31.115405 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 17:23:31.115764 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":163,"threads_started":1}
I20260923 17:23:31.118010 11706 heartbeater.cc:347] Connected to a master server at 127.23.214.190:40671
I20260923 17:23:31.118232 11706 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:31.118664 11706 heartbeater.cc:511] Master 127.23.214.190:40671 requested a full tablet report, sending...
I20260923 17:23:31.119764 11242 ts_manager.cc:194] Registered new tserver with Master: b280d60db15e4088816cb1d20c61c931 (127.23.214.131:36511)
I20260923 17:23:31.120337 11242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.214.131@KRBTEST.COM'} at 127.23.214.131:56449
I20260923 17:23:31.128726 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:23:31.138389 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.130742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.23.214.190:40671)
0923 17:23:31.130901 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.130906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.131095 (+ 189us) server_negotiation.cc:407] Connection header received
0923 17:23:31.131254 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.131256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.131315 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.131395 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:31.132103 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.132632 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.133516 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.133781 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.133927 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.135278 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:31.135282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:31.135284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:31.135318 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:31.136618 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:31.137046 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:31.137050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:31.137052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:31.137137 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:31.137399 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:31.137403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:31.137404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:31.137622 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:31.137788 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.138094 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.138208 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:23:31.141081 11242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:23:31.144196 11242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:23:31.159178 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.153504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:34903 (local address 127.0.0.1:50048)
0923 17:23:31.154479 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:31.154491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:31.154584 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:31.155061 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:31.155064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:31.155080 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:31.155163 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.155169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.156544 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.156548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.157695 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.157704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.158646 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.158650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.158756 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.158792 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:23:31.158910 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:23:31.159032 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":5,"thread_start_us":119,"threads_started":1}
I20260923 17:23:31.159184 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.153650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.23.214.130:34903)
0923 17:23:31.154482 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.154495 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.154559 (+ 64us) server_negotiation.cc:407] Connection header received
0923 17:23:31.154723 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.154729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.154925 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.155089 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:31.155263 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.156397 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.157815 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.158521 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.158661 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.158793 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.158945 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.159032 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":153,"threads_started":1}
I20260923 17:23:31.159310 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.152903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.23.214.129:40811)
0923 17:23:31.153213 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.153235 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.154036 (+ 801us) server_negotiation.cc:407] Connection header received
0923 17:23:31.154278 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.154284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.154518 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.154699 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:31.155000 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.156000 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.157585 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.158474 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.158568 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.158690 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.159067 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.159163 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":119,"threads_started":1}
I20260923 17:23:31.159711 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.153844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:36511 (local address 127.0.0.1:38604)
0923 17:23:31.154753 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:31.154764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:31.154828 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:31.155473 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:31.155476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:31.155493 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:31.155575 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.155580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.156905 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.156908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.158016 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.158026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.159245 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.159248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.159354 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.159378 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:23:31.159452 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:23:31.159572 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":57,"thread_start_us":134,"threads_started":1}
I20260923 17:23:31.159751 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.154053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.23.214.131:36511)
0923 17:23:31.155002 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.155016 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.155040 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:23:31.155111 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.155116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.155332 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.155485 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:31.155686 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.156753 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.158143 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.158957 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.159072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.159185 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.159537 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.159629 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":110,"threads_started":1}
I20260923 17:23:31.159714 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.152764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:40811 (local address 127.0.0.1:40654)
0923 17:23:31.153896 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:31.153950 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:31.154088 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:31.154678 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:31.154686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:31.154706 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:31.154874 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.154882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.156124 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.156130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.157452 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.157462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.158803 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.158807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.158899 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.158923 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:23:31.159452 (+ 529us) client_negotiation.cc:240] Negotiation successful
0923 17:23:31.159579 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":50,"thread_start_us":93,"threads_started":1}
I20260923 17:23:31.162879 11639 tablet_service.cc:1467] Processing CreateTablet for tablet 639bd62494ef4d0d811ff2a2f25cbe37 (DEFAULT_TABLE table=test-table [id=693266b0b12646c383f49c06c6f7ba91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:31.162894 11504 tablet_service.cc:1467] Processing CreateTablet for tablet 639bd62494ef4d0d811ff2a2f25cbe37 (DEFAULT_TABLE table=test-table [id=693266b0b12646c383f49c06c6f7ba91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:31.162894 11369 tablet_service.cc:1467] Processing CreateTablet for tablet 639bd62494ef4d0d811ff2a2f25cbe37 (DEFAULT_TABLE table=test-table [id=693266b0b12646c383f49c06c6f7ba91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:31.163990 11504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 639bd62494ef4d0d811ff2a2f25cbe37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:31.163990 11369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 639bd62494ef4d0d811ff2a2f25cbe37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:31.163993 11639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 639bd62494ef4d0d811ff2a2f25cbe37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:31.171334 11723 tablet_bootstrap.cc:492] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Bootstrap starting.
I20260923 17:23:31.172219 11725 tablet_bootstrap.cc:492] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Bootstrap starting.
I20260923 17:23:31.174201 11724 tablet_bootstrap.cc:492] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: Bootstrap starting.
I20260923 17:23:31.174242 11725 tablet_bootstrap.cc:654] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:31.174242 11723 tablet_bootstrap.cc:654] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:31.175614 11725 log.cc:824] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:31.175985 11724 tablet_bootstrap.cc:654] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:31.176452 11723 log.cc:824] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:31.176697 11724 log.cc:824] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:31.177937 11725 tablet_bootstrap.cc:492] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: No bootstrap required, opened a new log
I20260923 17:23:31.178138 11725 ts_tablet_manager.cc:1397] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260923 17:23:31.178457 11723 tablet_bootstrap.cc:492] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: No bootstrap required, opened a new log
I20260923 17:23:31.178483 11724 tablet_bootstrap.cc:492] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: No bootstrap required, opened a new log
I20260923 17:23:31.178632 11723 ts_tablet_manager.cc:1397] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 17:23:31.178694 11724 ts_tablet_manager.cc:1397] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:23:31.182009 11724 raft_consensus.cc:348] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.182075 11725 raft_consensus.cc:348] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.182319 11725 raft_consensus.cc:374] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:31.182331 11724 raft_consensus.cc:374] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:31.182372 11725 raft_consensus.cc:729] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00b9a3ce11e464a891413de36a042e2, State: Initialized, Role: FOLLOWER
I20260923 17:23:31.182389 11724 raft_consensus.cc:729] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba4b3fa7d6e4cdc9318dfb56e25d66e, State: Initialized, Role: FOLLOWER
I20260923 17:23:31.182319 11723 raft_consensus.cc:348] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.182538 11723 raft_consensus.cc:374] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:31.182588 11723 raft_consensus.cc:729] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b280d60db15e4088816cb1d20c61c931, State: Initialized, Role: FOLLOWER
I20260923 17:23:31.182845 11724 consensus_queue.cc:260] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.182857 11725 consensus_queue.cc:260] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.183907 11724 ts_tablet_manager.cc:1428] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 17:23:31.183919 11725 ts_tablet_manager.cc:1428] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:23:31.184036 11436 heartbeater.cc:503] Master 127.23.214.190:40671 was elected leader, sending a full tablet report...
I20260923 17:23:31.184090 11723 consensus_queue.cc:260] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.184710 11706 heartbeater.cc:503] Master 127.23.214.190:40671 was elected leader, sending a full tablet report...
I20260923 17:23:31.184887 11723 ts_tablet_manager.cc:1428] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:23:31.185227 11571 heartbeater.cc:503] Master 127.23.214.190:40671 was elected leader, sending a full tablet report...
W20260923 17:23:31.352417 11707 tablet.cc:2369] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:23:31.371040 11572 tablet.cc:2369] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:23:31.398619 11437 tablet.cc:2369] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:31.432713 11730 raft_consensus.cc:482] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:31.432965 11730 raft_consensus.cc:504] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.434186 11730 leader_election.cc:288] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ba4b3fa7d6e4cdc9318dfb56e25d66e (127.23.214.129:40811), b280d60db15e4088816cb1d20c61c931 (127.23.214.131:36511)
I20260923 17:23:31.438723 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.434497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:40811 (local address 127.23.214.130:49571)
0923 17:23:31.434829 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:31.434842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:31.434979 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:31.435373 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:31.435376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:31.435394 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:31.435531 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.435541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.436481 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.436484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.437404 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.437414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.438245 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.438251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.438363 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.438396 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:23:31.438470 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:23:31.438556 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":22,"thread_start_us":154,"threads_started":1}
I20260923 17:23:31.438735 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.434596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:49571 (local address 127.23.214.129:40811)
0923 17:23:31.434746 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.434751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.434923 (+ 172us) server_negotiation.cc:407] Connection header received
0923 17:23:31.435166 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.435172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.435239 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.435343 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:31.435703 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.436363 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.437540 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.438115 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.438222 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.438302 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.438512 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.438596 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 17:23:31.438766 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.435145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:55451 (local address 127.23.214.131:36511)
0923 17:23:31.435272 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.435282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.435510 (+ 228us) server_negotiation.cc:407] Connection header received
0923 17:23:31.435612 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.435618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.435679 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.435747 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:31.436065 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.436593 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.437750 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.438199 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.438316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.438367 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.438576 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.438652 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:23:31.438891 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.435026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:36511 (local address 127.23.214.130:55451)
0923 17:23:31.435397 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:31.435412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:31.435524 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:31.435792 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:31.435795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:31.435814 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:31.435947 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.435954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.436711 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.436714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.437617 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:31.437628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.438336 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.438346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:31.438424 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.438442 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:23:31.438550 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:23:31.438671 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260923 17:23:31.439590 11659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639bd62494ef4d0d811ff2a2f25cbe37" candidate_uuid: "f00b9a3ce11e464a891413de36a042e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b280d60db15e4088816cb1d20c61c931" is_pre_election: true
I20260923 17:23:31.439634 11389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639bd62494ef4d0d811ff2a2f25cbe37" candidate_uuid: "f00b9a3ce11e464a891413de36a042e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" is_pre_election: true
I20260923 17:23:31.439913 11659 raft_consensus.cc:2454] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00b9a3ce11e464a891413de36a042e2 in term 0.
I20260923 17:23:31.439915 11389 raft_consensus.cc:2454] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00b9a3ce11e464a891413de36a042e2 in term 0.
I20260923 17:23:31.440409 11457 leader_election.cc:302] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b280d60db15e4088816cb1d20c61c931, f00b9a3ce11e464a891413de36a042e2; no voters:
I20260923 17:23:31.440724 11730 raft_consensus.cc:2781] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:23:31.440822 11730 raft_consensus.cc:482] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:23:31.440871 11730 raft_consensus.cc:3037] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:31.442143 11730 raft_consensus.cc:504] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.442544 11730 leader_election.cc:288] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [CANDIDATE]: Term 1 election: Requested vote from peers 1ba4b3fa7d6e4cdc9318dfb56e25d66e (127.23.214.129:40811), b280d60db15e4088816cb1d20c61c931 (127.23.214.131:36511)
I20260923 17:23:31.442958 11389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639bd62494ef4d0d811ff2a2f25cbe37" candidate_uuid: "f00b9a3ce11e464a891413de36a042e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e"
I20260923 17:23:31.443020 11659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639bd62494ef4d0d811ff2a2f25cbe37" candidate_uuid: "f00b9a3ce11e464a891413de36a042e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b280d60db15e4088816cb1d20c61c931"
I20260923 17:23:31.443090 11389 raft_consensus.cc:3037] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:31.443172 11659 raft_consensus.cc:3037] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:31.444175 11389 raft_consensus.cc:2454] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00b9a3ce11e464a891413de36a042e2 in term 1.
I20260923 17:23:31.444355 11659 raft_consensus.cc:2454] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00b9a3ce11e464a891413de36a042e2 in term 1.
I20260923 17:23:31.444507 11457 leader_election.cc:302] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba4b3fa7d6e4cdc9318dfb56e25d66e, f00b9a3ce11e464a891413de36a042e2; no voters:
I20260923 17:23:31.444677 11730 raft_consensus.cc:2781] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:31.444917 11730 raft_consensus.cc:686] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 1 LEADER]: Becoming Leader. State: Replica: f00b9a3ce11e464a891413de36a042e2, State: Running, Role: LEADER
I20260923 17:23:31.445267 11730 consensus_queue.cc:237] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } }
I20260923 17:23:31.448539 11242 catalog_manager.cc:5495] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f00b9a3ce11e464a891413de36a042e2 (127.23.214.130). New cstate: current_term: 1 leader_uuid: "f00b9a3ce11e464a891413de36a042e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f00b9a3ce11e464a891413de36a042e2" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 } health_report { overall_health: UNKNOWN } } }
I20260923 17:23:31.521377 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.518031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.23.214.130:34903)
0923 17:23:31.518160 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.518164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.518177 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:23:31.518324 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.518327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.518386 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.518477 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:23:31.518696 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.519215 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.520095 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.520306 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.520381 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.520640 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:23:31.521058 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:23:31.521162 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.521217 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.521266 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:23:31.527654 11659 raft_consensus.cc:1261] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 1 FOLLOWER]: Refusing update from remote peer f00b9a3ce11e464a891413de36a042e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:31.527654 11389 raft_consensus.cc:1261] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 1 FOLLOWER]: Refusing update from remote peer f00b9a3ce11e464a891413de36a042e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:31.528426 11734 consensus_queue.cc:1035] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba4b3fa7d6e4cdc9318dfb56e25d66e" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 40811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:31.528658 11730 consensus_queue.cc:1035] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b280d60db15e4088816cb1d20c61c931" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 36511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:31.536926 11738 mvcc.cc:205] Tried to move back new op lower bound from 7332594530408820736 to 7332594530082639872. Current Snapshot: MvccSnapshot[applied={T|T < 7332594530408820736}]
I20260923 17:23:31.537992 11737 mvcc.cc:205] Tried to move back new op lower bound from 7332594530408820736 to 7332594530082639872. Current Snapshot: MvccSnapshot[applied={T|T < 7332594530408820736 or (T in {7332594530408820736})}]
I20260923 17:23:31.544070 11242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:31.544221 11242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:31.546592 11242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b477c80581834e44a258f82337c991a8: Sending DeleteTablet for 3 replicas of tablet 639bd62494ef4d0d811ff2a2f25cbe37
I20260923 17:23:31.547422 11504 tablet_service.cc:1514] Processing DeleteTablet for tablet 639bd62494ef4d0d811ff2a2f25cbe37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:31 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:50048
I20260923 17:23:31.547412 11369 tablet_service.cc:1514] Processing DeleteTablet for tablet 639bd62494ef4d0d811ff2a2f25cbe37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:31 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:40654
I20260923 17:23:31.548046 11747 tablet_replica.cc:330] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: stopping tablet replica
I20260923 17:23:31.548046 11746 tablet_replica.cc:330] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e: stopping tablet replica
I20260923 17:23:31.548373 11747 raft_consensus.cc:2229] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:23:31.548372 11746 raft_consensus.cc:2229] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:31.548588 11639 tablet_service.cc:1514] Processing DeleteTablet for tablet 639bd62494ef4d0d811ff2a2f25cbe37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:31 UTC) from {username='kudu', principal='kudu/127.23.214.190@KRBTEST.COM'} at 127.0.0.1:38604
I20260923 17:23:31.548712 11747 raft_consensus.cc:2258] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:31.548883 11746 raft_consensus.cc:2258] T 639bd62494ef4d0d811ff2a2f25cbe37 P 1ba4b3fa7d6e4cdc9318dfb56e25d66e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:31.549399 11748 tablet_replica.cc:330] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: stopping tablet replica
I20260923 17:23:31.549562 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11306
I20260923 17:23:31.550720 11748 raft_consensus.cc:2229] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:31.551011 11748 raft_consensus.cc:2258] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:31.552623 11748 ts_tablet_manager.cc:1905] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:31.552953 11747 ts_tablet_manager.cc:1905] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:31.555406 11748 ts_tablet_manager.cc:1918] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:31.555503 11748 log.cc:1196] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/639bd62494ef4d0d811ff2a2f25cbe37
I20260923 17:23:31.555821 11748 ts_tablet_manager.cc:1939] T 639bd62494ef4d0d811ff2a2f25cbe37 P b280d60db15e4088816cb1d20c61c931: Deleting consensus metadata
I20260923 17:23:31.556268 11747 ts_tablet_manager.cc:1918] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:31.556380 11747 log.cc:1196] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/639bd62494ef4d0d811ff2a2f25cbe37
I20260923 17:23:31.556687 11747 ts_tablet_manager.cc:1939] T 639bd62494ef4d0d811ff2a2f25cbe37 P f00b9a3ce11e464a891413de36a042e2: Deleting consensus metadata
I20260923 17:23:31.557129 11225 catalog_manager.cc:4841] TS b280d60db15e4088816cb1d20c61c931 (127.23.214.131:36511): tablet 639bd62494ef4d0d811ff2a2f25cbe37 (table test-table [id=693266b0b12646c383f49c06c6f7ba91]) successfully deleted
W20260923 17:23:31.558131 11225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:40811 (error 108)
I20260923 17:23:31.559031 11225 catalog_manager.cc:4841] TS f00b9a3ce11e464a891413de36a042e2 (127.23.214.130:34903): tablet 639bd62494ef4d0d811ff2a2f25cbe37 (table test-table [id=693266b0b12646c383f49c06c6f7ba91]) successfully deleted
I20260923 17:23:31.559108 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11441
I20260923 17:23:31.559381 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.558892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:40811 (local address 127.0.0.1:40658)
0923 17:23:31.559249 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:31.559308 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
W20260923 17:23:31.560478 11225 catalog_manager.cc:4584] TS 1ba4b3fa7d6e4cdc9318dfb56e25d66e (127.23.214.129:40811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 639bd62494ef4d0d811ff2a2f25cbe37: Network error: Client connection negotiation failed: client connection to 127.23.214.129:40811: connect: Connection refused (error 111)
I20260923 17:23:31.565011 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11576
I20260923 17:23:31.570609 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11209
2026-09-23T17:23:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2540 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T17:23:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:31Z Disabled control of system clock
I20260923 17:23:31.601004 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38235
--webserver_interface=127.23.214.190
--webserver_port=0
--builtin_ntp_servers=127.23.214.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:23:31.711369 11753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:31.711637 11753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:31.711674 11753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:31.715497 11753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:31.715543 11753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:31.715557 11753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:31.715574 11753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:31.715588 11753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:31.720247 11753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:38235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:31.721208 11753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:31.722575 11753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:23:31.730909 11753 server_base.cc:1034] running on GCE node
W20260923 17:23:31.730877 11758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:31.730870 11761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:31.730890 11759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:31.731680 11753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:31.732721 11753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:31.733940 11753 hybrid_clock.cc:648] HybridClock initialized: now 1790184211733900 us; error 42 us; skew 500 ppm
I20260923 17:23:31.735785 11753 webserver.cc:460] Webserver started at http://127.23.214.190:46705/ using document root <none> and password file <none>
I20260923 17:23:31.736423 11753 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:31.736516 11753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:31.736780 11753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:31.738871 11753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "c2d232c55dbc458db721e1a101d0729c"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "411bdcd5a4e6b50fa0bd561d5e0a119b"
server_key_iv: "e955f72dc270caa9c9080c66e44b0f39"
server_key_version: "encryptionkey@0"
I20260923 17:23:31.739418 11753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "c2d232c55dbc458db721e1a101d0729c"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "411bdcd5a4e6b50fa0bd561d5e0a119b"
server_key_iv: "e955f72dc270caa9c9080c66e44b0f39"
server_key_version: "encryptionkey@0"
I20260923 17:23:31.743065 11753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:23:31.745407 11767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:31.746737 11753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:31.746871 11753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "c2d232c55dbc458db721e1a101d0729c"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "411bdcd5a4e6b50fa0bd561d5e0a119b"
server_key_iv: "e955f72dc270caa9c9080c66e44b0f39"
server_key_version: "encryptionkey@0"
I20260923 17:23:31.746982 11753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:31.771293 11753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:31.771992 11753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:31.772197 11753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:31.779413 11753 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:38235
I20260923 17:23:31.779418 11819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:38235 every 8 connection(s)
I20260923 17:23:31.780450 11753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:31.783555 11820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:31.787066 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11753
I20260923 17:23:31.787222 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:31.787505 24410 external_mini_cluster.cc:1428] Setting key 6b31f6ff8ecc9f258a977c3774203bb1
I20260923 17:23:31.789361 11820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c: Bootstrap starting.
I20260923 17:23:31.791746 11820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:31.792505 11820 log.cc:824] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:31.794641 11820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c: No bootstrap required, opened a new log
I20260923 17:23:31.796557 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:31.788926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.23.214.190:38235)
0923 17:23:31.789575 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:31.789592 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:31.789627 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:23:31.790307 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:31.790324 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:31.790718 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:31.790991 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:31.791327 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.792340 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.793447 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:31.793950 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:31.794066 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:31.794885 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:31.794890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:31.794903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:31.795269 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:31.795332 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:31.795351 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:31.795422 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:31.795582 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 17:23:31.795876 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":137,"threads_started":1}
I20260923 17:23:31.798290 11820 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d232c55dbc458db721e1a101d0729c" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38235 } }
I20260923 17:23:31.798493 11820 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:31.798570 11820 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d232c55dbc458db721e1a101d0729c, State: Initialized, Role: FOLLOWER
I20260923 17:23:31.798993 11820 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d232c55dbc458db721e1a101d0729c" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38235 } }
I20260923 17:23:31.799139 11820 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:31.799213 11820 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:31.799340 11820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:31.800308 11820 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d232c55dbc458db721e1a101d0729c" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38235 } }
I20260923 17:23:31.800640 11820 leader_election.cc:302] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d232c55dbc458db721e1a101d0729c; no voters:
I20260923 17:23:31.800932 11820 leader_election.cc:288] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:31.801048 11825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:31.801281 11825 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [term 1 LEADER]: Becoming Leader. State: Replica: c2d232c55dbc458db721e1a101d0729c, State: Running, Role: LEADER
I20260923 17:23:31.801571 11825 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d232c55dbc458db721e1a101d0729c" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38235 } }
I20260923 17:23:31.802167 11820 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:31.803251 11826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d232c55dbc458db721e1a101d0729c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d232c55dbc458db721e1a101d0729c" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38235 } } }
I20260923 17:23:31.803251 11827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d232c55dbc458db721e1a101d0729c. Latest consensus state: current_term: 1 leader_uuid: "c2d232c55dbc458db721e1a101d0729c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d232c55dbc458db721e1a101d0729c" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38235 } } }
I20260923 17:23:31.803416 11826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:31.803416 11827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:31.803866 11834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:31.807008 11834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:31.812434 11834 catalog_manager.cc:1347] Generated new cluster ID: 26369dab58204afe91196875ef507b6e
I20260923 17:23:31.812498 11834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:31.827284 11834 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:31.828269 11834 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:31.836175 11834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c: Generated new TSK 0
I20260923 17:23:31.837005 11834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:31.850672 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.129:0
--local_ip_for_outbound_sockets=127.23.214.129
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38235
--builtin_ntp_servers=127.23.214.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:23:31.967235 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:31.967557 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:31.967629 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:31.971692 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:31.971796 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:31.971889 11844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.129
I20260923 17:23:31.976735 11844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.214.129
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.129
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:31.977931 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:31.979393 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:31.989315 11850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:31.989272 11852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:31.989272 11849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:31.989576 11844 server_base.cc:1034] running on GCE node
I20260923 17:23:31.990036 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:31.990693 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:31.991848 11844 hybrid_clock.cc:648] HybridClock initialized: now 1790184211991819 us; error 43 us; skew 500 ppm
I20260923 17:23:31.993636 11844 webserver.cc:460] Webserver started at http://127.23.214.129:39505/ using document root <none> and password file <none>
I20260923 17:23:31.994253 11844 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:31.994323 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:31.994555 11844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:31.996580 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data/instance:
uuid: "db530a69230e4ed1a3f373a7f6dcccc1"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "c3dfa6cba761be2d1fd0b2e43d49829d"
server_key_iv: "8b0a1a85ca645b78f2db1b60b94ecee6"
server_key_version: "encryptionkey@0"
I20260923 17:23:31.997135 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance:
uuid: "db530a69230e4ed1a3f373a7f6dcccc1"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "c3dfa6cba761be2d1fd0b2e43d49829d"
server_key_iv: "8b0a1a85ca645b78f2db1b60b94ecee6"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.000821 11844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:23:32.003374 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.004539 11844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:32.004659 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/wal
uuid: "db530a69230e4ed1a3f373a7f6dcccc1"
format_stamp: "Formatted at 2026-09-23 17:23:31 on dist-test-slave-9q7f"
server_key: "c3dfa6cba761be2d1fd0b2e43d49829d"
server_key_iv: "8b0a1a85ca645b78f2db1b60b94ecee6"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.004761 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:32.034284 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:32.035023 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:32.035236 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:32.035871 11844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:32.036845 11844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:32.036892 11844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.036957 11844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:32.036988 11844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.046115 11844 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.129:42583
I20260923 17:23:32.046132 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.129:42583 every 8 connection(s)
I20260923 17:23:32.047107 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/data/info.pb
I20260923 17:23:32.050866 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11844
I20260923 17:23:32.050992 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-0/wal/instance
I20260923 17:23:32.051275 24410 external_mini_cluster.cc:1428] Setting key e9f58ce18d4b940735fa98ce1763a8b7
I20260923 17:23:32.053247 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.130:0
--local_ip_for_outbound_sockets=127.23.214.130
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38235
--builtin_ntp_servers=127.23.214.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:23:32.057152 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.049072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.129:58713 (local address 127.23.214.190:38235)
0923 17:23:32.049192 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.049196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.050324 (+ 1128us) server_negotiation.cc:407] Connection header received
0923 17:23:32.052071 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.052075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.052142 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.052219 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:32.053299 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.053970 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.055134 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.055382 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.055498 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.056639 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:32.056643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:32.056646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:32.056666 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:32.056684 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:32.056692 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:32.056769 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.056929 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.057004 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:23:32.058096 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.049476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38235 (local address 127.23.214.129:58713)
0923 17:23:32.050171 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.050213 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.051848 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.052404 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.052411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.052498 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:32.053157 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.053174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.054097 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.054100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.055007 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.055016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.055537 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.055541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.055716 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.056383 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:32.056384 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:32.056425 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:32.056490 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:32.056782 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:32.056795 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:32.056805 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.056930 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.057224 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260923 17:23:32.059396 11972 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38235
I20260923 17:23:32.059653 11972 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:32.060190 11972 heartbeater.cc:511] Master 127.23.214.190:38235 requested a full tablet report, sending...
I20260923 17:23:32.061957 11784 ts_manager.cc:194] Registered new tserver with Master: db530a69230e4ed1a3f373a7f6dcccc1 (127.23.214.129:42583)
I20260923 17:23:32.063772 11784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.129:58713
W20260923 17:23:32.167330 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:32.167588 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:32.167627 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:32.171763 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:32.171819 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:32.171876 11975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.130
I20260923 17:23:32.176626 11975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.214.130
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.130
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:32.177711 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:32.179214 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:23:32.189085 11975 server_base.cc:1034] running on GCE node
W20260923 17:23:32.189054 11981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:32.189097 11980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:32.189028 11983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:32.189673 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:32.190368 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:32.191527 11975 hybrid_clock.cc:648] HybridClock initialized: now 1790184212191511 us; error 30 us; skew 500 ppm
I20260923 17:23:32.193336 11975 webserver.cc:460] Webserver started at http://127.23.214.130:34191/ using document root <none> and password file <none>
I20260923 17:23:32.193996 11975 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:32.194063 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:32.194308 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:32.196446 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data/instance:
uuid: "e8485bea024544e9bac4cb49323afeca"
format_stamp: "Formatted at 2026-09-23 17:23:32 on dist-test-slave-9q7f"
server_key: "b224e67f2e00e7c4225935c5a0bc5acc"
server_key_iv: "602cf36746a6ff6607cc4ba7b3bd0562"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.196995 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance:
uuid: "e8485bea024544e9bac4cb49323afeca"
format_stamp: "Formatted at 2026-09-23 17:23:32 on dist-test-slave-9q7f"
server_key: "b224e67f2e00e7c4225935c5a0bc5acc"
server_key_iv: "602cf36746a6ff6607cc4ba7b3bd0562"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.200603 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260923 17:23:32.203123 11989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.204388 11975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:32.204512 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal
uuid: "e8485bea024544e9bac4cb49323afeca"
format_stamp: "Formatted at 2026-09-23 17:23:32 on dist-test-slave-9q7f"
server_key: "b224e67f2e00e7c4225935c5a0bc5acc"
server_key_iv: "602cf36746a6ff6607cc4ba7b3bd0562"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.204618 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:32.265468 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:32.266316 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:32.266536 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:32.267165 11975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:32.268177 11975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:32.268254 11975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.268330 11975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:32.268375 11975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.277659 11975 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.130:35255
I20260923 17:23:32.277684 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.130:35255 every 8 connection(s)
I20260923 17:23:32.278684 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/data/info.pb
I20260923 17:23:32.280876 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 11975
I20260923 17:23:32.281025 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal/instance
I20260923 17:23:32.281339 24410 external_mini_cluster.cc:1428] Setting key 980ecc55042acdee08731fef8a9670e6
I20260923 17:23:32.283265 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.214.131:0
--local_ip_for_outbound_sockets=127.23.214.131
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38235
--builtin_ntp_servers=127.23.214.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:23:32.288213 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.280489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:57471 (local address 127.23.214.190:38235)
0923 17:23:32.280765 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.280772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.281826 (+ 1054us) server_negotiation.cc:407] Connection header received
0923 17:23:32.283403 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.283406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.283459 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.283537 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:32.284460 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.284998 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.286205 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.286432 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.286552 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.287689 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:32.287698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:32.287700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:32.287711 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:32.287730 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:32.287744 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:32.287821 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.288004 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.288065 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260923 17:23:32.289093 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.280881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38235 (local address 127.23.214.130:57471)
0923 17:23:32.281652 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.281704 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.283186 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.283707 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.283714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.283804 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:32.284315 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.284328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.285131 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.285140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.286083 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.286092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.286587 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.286591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.286755 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.287425 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:32.287426 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:32.287477 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:32.287539 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:32.287821 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:32.287832 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:32.287840 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.287987 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.288281 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":159,"threads_started":1}
I20260923 17:23:32.290460 12103 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38235
I20260923 17:23:32.290697 12103 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:32.291208 12103 heartbeater.cc:511] Master 127.23.214.190:38235 requested a full tablet report, sending...
I20260923 17:23:32.292325 11784 ts_manager.cc:194] Registered new tserver with Master: e8485bea024544e9bac4cb49323afeca (127.23.214.130:35255)
I20260923 17:23:32.292903 11784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.130:57471
W20260923 17:23:32.400959 12106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:32.401293 12106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:32.401389 12106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:32.405841 12106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:32.405961 12106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:32.406050 12106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.214.131
I20260923 17:23:32.411286 12106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.214.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.214.131
--webserver_port=0
--tserver_master_addrs=127.23.214.190:38235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.214.131
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:32.412545 12106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:32.414029 12106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:32.423815 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:32.423825 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:32.423815 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:32.424185 12106 server_base.cc:1034] running on GCE node
I20260923 17:23:32.424612 12106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:32.425199 12106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:32.426364 12106 hybrid_clock.cc:648] HybridClock initialized: now 1790184212426349 us; error 26 us; skew 500 ppm
I20260923 17:23:32.428180 12106 webserver.cc:460] Webserver started at http://127.23.214.131:35191/ using document root <none> and password file <none>
I20260923 17:23:32.428783 12106 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:32.428848 12106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:32.429101 12106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:32.431257 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data/instance:
uuid: "739b1d35bce848b292ce1d89c5b47d89"
format_stamp: "Formatted at 2026-09-23 17:23:32 on dist-test-slave-9q7f"
server_key: "310db59d3c0eaadf14b5e0d0d018a787"
server_key_iv: "3a4ecc068886dff14d29bc135b7446e7"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.431819 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance:
uuid: "739b1d35bce848b292ce1d89c5b47d89"
format_stamp: "Formatted at 2026-09-23 17:23:32 on dist-test-slave-9q7f"
server_key: "310db59d3c0eaadf14b5e0d0d018a787"
server_key_iv: "3a4ecc068886dff14d29bc135b7446e7"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.435576 12106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 17:23:32.438221 12120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.439656 12106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:23:32.439821 12106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal
uuid: "739b1d35bce848b292ce1d89c5b47d89"
format_stamp: "Formatted at 2026-09-23 17:23:32 on dist-test-slave-9q7f"
server_key: "310db59d3c0eaadf14b5e0d0d018a787"
server_key_iv: "3a4ecc068886dff14d29bc135b7446e7"
server_key_version: "encryptionkey@0"
I20260923 17:23:32.439945 12106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:32.456727 12106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:32.457432 12106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:32.457645 12106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:32.458343 12106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:23:32.459309 12106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:23:32.459386 12106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.459458 12106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:23:32.459510 12106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:32.468295 12106 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.131:43851
I20260923 17:23:32.468320 12235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.131:43851 every 8 connection(s)
I20260923 17:23:32.469585 12106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/data/info.pb
I20260923 17:23:32.470798 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12106
I20260923 17:23:32.470916 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal/instance
I20260923 17:23:32.471268 24410 external_mini_cluster.cc:1428] Setting key 1b279fb7162480f53e9fcafafa328dad
I20260923 17:23:32.481357 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.473381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.131:58539 (local address 127.23.214.190:38235)
0923 17:23:32.473485 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.473489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.473502 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:23:32.475150 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.475154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.475206 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.475253 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:32.476558 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.477369 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.479065 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.479494 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.479599 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.480837 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:32.480840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:32.480843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:32.480858 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:32.480886 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:32.480894 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:32.480978 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.481141 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.481205 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 17:23:32.482350 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.472359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:38235 (local address 127.23.214.131:58539)
0923 17:23:32.473149 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.473204 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.474906 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.475648 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.475656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.475763 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:32.476397 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.476415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.477532 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.477535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.478909 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.478920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.479655 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.479660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.479849 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.480519 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:23:32.480521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:32.480597 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:23:32.480680 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:32.480978 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:32.480988 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:32.480996 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.481142 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.481440 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260923 17:23:32.483492 12236 heartbeater.cc:347] Connected to a master server at 127.23.214.190:38235
I20260923 17:23:32.483713 12236 heartbeater.cc:464] Registering TS with master...
I20260923 17:23:32.484351 12236 heartbeater.cc:511] Master 127.23.214.190:38235 requested a full tablet report, sending...
I20260923 17:23:32.485409 11784 ts_manager.cc:194] Registered new tserver with Master: 739b1d35bce848b292ce1d89c5b47d89 (127.23.214.131:43851)
I20260923 17:23:32.486053 11784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.214.131:58539
I20260923 17:23:32.497397 24410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:23:32.503638 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.499444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.23.214.190:38235)
0923 17:23:32.499595 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.499599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.499790 (+ 191us) server_negotiation.cc:407] Connection header received
0923 17:23:32.499963 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.499967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.500016 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.500102 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:32.500389 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.500918 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.501778 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.502160 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.502287 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.503137 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:32.503140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:32.503142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:32.503153 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:32.503171 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:32.503179 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:32.503257 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.503396 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.503447 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:23:32.506356 11784 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:23:32.509727 11784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:23:32.524034 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.518690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.23.214.131:43851)
0923 17:23:32.519468 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.519479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.519498 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:23:32.519565 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.519571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.519736 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.520148 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:32.520271 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.520987 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.522810 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.523331 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.523474 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.523656 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.523758 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.523853 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":158,"threads_started":1}
I20260923 17:23:32.524029 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.517998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:43851 (local address 127.0.0.1:59932)
0923 17:23:32.518698 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.518746 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.518869 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.519880 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.519883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.519899 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:32.520002 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.520009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.521110 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.521116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.522654 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.522665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.523458 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.523462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.523571 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.523608 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.523710 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.523828 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":88,"thread_start_us":76,"threads_started":1}
I20260923 17:23:32.524220 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.518078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42583 (local address 127.0.0.1:57458)
0923 17:23:32.519100 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.519116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.519223 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.519712 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.519717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.519740 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:32.519895 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.519905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.521394 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.521398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.522546 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.522555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.523856 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.523860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.523955 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.523977 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.524039 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.524106 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":61,"thread_start_us":108,"threads_started":1}
I20260923 17:23:32.524056 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.518707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.130:35255 (local address 127.0.0.1:39552)
0923 17:23:32.519177 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.519189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.519265 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.519945 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.519947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.519959 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:32.520022 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.520028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.521357 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.521361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.522654 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.522665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.523632 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.523636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.523718 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.523741 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.523792 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.523866 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":66,"thread_start_us":110,"threads_started":1}
I20260923 17:23:32.524618 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.518239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.23.214.129:42583)
0923 17:23:32.518512 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.518519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.519198 (+ 679us) server_negotiation.cc:407] Connection header received
0923 17:23:32.519342 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.519347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.519557 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.519721 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:32.520029 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.521247 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.522669 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.523481 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.523619 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.523787 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.524385 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.524472 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260923 17:23:32.525580 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.518894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.23.214.130:35255)
0923 17:23:32.519517 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.519531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.519553 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:23:32.519631 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.519638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.519825 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.519985 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:32.520115 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.521235 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.522778 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.523506 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.524136 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.524311 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.524364 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.524454 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":85,"threads_started":1}
I20260923 17:23:32.527680 12037 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0aedbd58c547ce887c6e18b73f62ed (DEFAULT_TABLE table=test-table [id=5f1a745f437c4163a62e8f7da0797ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:32.527690 11906 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0aedbd58c547ce887c6e18b73f62ed (DEFAULT_TABLE table=test-table [id=5f1a745f437c4163a62e8f7da0797ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:32.527691 12168 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0aedbd58c547ce887c6e18b73f62ed (DEFAULT_TABLE table=test-table [id=5f1a745f437c4163a62e8f7da0797ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:23:32.528645 12037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0aedbd58c547ce887c6e18b73f62ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:32.528663 12168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0aedbd58c547ce887c6e18b73f62ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:32.528664 11906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0aedbd58c547ce887c6e18b73f62ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:32.534538 12253 tablet_bootstrap.cc:492] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Bootstrap starting.
I20260923 17:23:32.536142 12254 tablet_bootstrap.cc:492] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1: Bootstrap starting.
I20260923 17:23:32.536461 12255 tablet_bootstrap.cc:492] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Bootstrap starting.
I20260923 17:23:32.537097 12253 tablet_bootstrap.cc:654] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:32.537967 12253 log.cc:824] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:32.538285 12254 tablet_bootstrap.cc:654] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:32.538601 12255 tablet_bootstrap.cc:654] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:32.539124 12254 log.cc:824] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:32.539335 12255 log.cc:824] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:32.540017 12253 tablet_bootstrap.cc:492] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: No bootstrap required, opened a new log
I20260923 17:23:32.540243 12253 ts_tablet_manager.cc:1397] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:23:32.540930 12254 tablet_bootstrap.cc:492] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1: No bootstrap required, opened a new log
I20260923 17:23:32.541139 12254 ts_tablet_manager.cc:1397] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:23:32.541296 12255 tablet_bootstrap.cc:492] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: No bootstrap required, opened a new log
I20260923 17:23:32.541473 12255 ts_tablet_manager.cc:1397] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:23:32.544638 12254 raft_consensus.cc:348] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.544993 12254 raft_consensus.cc:374] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:32.545058 12254 raft_consensus.cc:729] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db530a69230e4ed1a3f373a7f6dcccc1, State: Initialized, Role: FOLLOWER
I20260923 17:23:32.545523 12254 consensus_queue.cc:260] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.545684 12255 raft_consensus.cc:348] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.545686 12253 raft_consensus.cc:348] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.545959 12253 raft_consensus.cc:374] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:32.545959 12255 raft_consensus.cc:374] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:32.546020 12253 raft_consensus.cc:729] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8485bea024544e9bac4cb49323afeca, State: Initialized, Role: FOLLOWER
I20260923 17:23:32.546027 12255 raft_consensus.cc:729] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739b1d35bce848b292ce1d89c5b47d89, State: Initialized, Role: FOLLOWER
I20260923 17:23:32.546383 11972 heartbeater.cc:503] Master 127.23.214.190:38235 was elected leader, sending a full tablet report...
I20260923 17:23:32.546463 12253 consensus_queue.cc:260] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.546463 12255 consensus_queue.cc:260] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.546612 12254 ts_tablet_manager.cc:1428] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 17:23:32.547180 12103 heartbeater.cc:503] Master 127.23.214.190:38235 was elected leader, sending a full tablet report...
I20260923 17:23:32.547410 12253 ts_tablet_manager.cc:1428] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 17:23:32.547880 12236 heartbeater.cc:503] Master 127.23.214.190:38235 was elected leader, sending a full tablet report...
W20260923 17:23:32.548204 11973 tablet.cc:2369] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:32.549178 12255 ts_tablet_manager.cc:1428] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260923 17:23:32.720777 12237 tablet.cc:2369] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:23:32.779867 12104 tablet.cc:2369] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:23:32.814899 12260 raft_consensus.cc:482] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:23:32.815179 12260 raft_consensus.cc:504] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.816329 12260 leader_election.cc:288] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 739b1d35bce848b292ce1d89c5b47d89 (127.23.214.131:43851), db530a69230e4ed1a3f373a7f6dcccc1 (127.23.214.129:42583)
I20260923 17:23:32.820894 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.816883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:45983 (local address 127.23.214.131:43851)
0923 17:23:32.817025 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.817030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.817300 (+ 270us) server_negotiation.cc:407] Connection header received
0923 17:23:32.817461 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.817465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.817520 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.817587 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:32.818020 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.818640 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.819888 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.820276 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.820397 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.820464 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.820689 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.820764 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:23:32.820919 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.816768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.131:43851 (local address 127.23.214.130:45983)
0923 17:23:32.817209 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.817223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.817358 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.817682 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.817685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.817710 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:32.817855 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.817863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.818758 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.818762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.819723 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.819733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.820413 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.820419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.820518 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.820554 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.820668 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.820745 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":79,"thread_start_us":104,"threads_started":1}
I20260923 17:23:32.820992 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.816941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.214.130:38865 (local address 127.23.214.129:42583)
0923 17:23:32.817119 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.817126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.817223 (+ 97us) server_negotiation.cc:407] Connection header received
0923 17:23:32.817408 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.817413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.817466 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.817545 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:23:32.817994 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.818745 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.819837 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.820290 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.820462 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.820520 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.820775 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.820852 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 17:23:32.821249 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.816768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42583 (local address 127.23.214.130:38865)
0923 17:23:32.817116 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.817131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:32.817284 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:32.817614 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:32.817618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:32.817638 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:23:32.817823 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.817831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.818862 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.818865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.819705 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:32.819713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.820543 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.820547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:32.820648 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.820666 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:23:32.820756 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:23:32.820859 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":4,"thread_start_us":130,"threads_started":1}
I20260923 17:23:32.821743 12189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0aedbd58c547ce887c6e18b73f62ed" candidate_uuid: "e8485bea024544e9bac4cb49323afeca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739b1d35bce848b292ce1d89c5b47d89" is_pre_election: true
I20260923 17:23:32.821823 11926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0aedbd58c547ce887c6e18b73f62ed" candidate_uuid: "e8485bea024544e9bac4cb49323afeca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" is_pre_election: true
I20260923 17:23:32.822109 12189 raft_consensus.cc:2454] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8485bea024544e9bac4cb49323afeca in term 0.
I20260923 17:23:32.822161 11926 raft_consensus.cc:2454] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8485bea024544e9bac4cb49323afeca in term 0.
I20260923 17:23:32.822654 11992 leader_election.cc:302] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db530a69230e4ed1a3f373a7f6dcccc1, e8485bea024544e9bac4cb49323afeca; no voters:
I20260923 17:23:32.822940 12260 raft_consensus.cc:2781] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:23:32.823025 12260 raft_consensus.cc:482] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:23:32.823067 12260 raft_consensus.cc:3037] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:32.824057 12260 raft_consensus.cc:504] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.824376 12260 leader_election.cc:288] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [CANDIDATE]: Term 1 election: Requested vote from peers 739b1d35bce848b292ce1d89c5b47d89 (127.23.214.131:43851), db530a69230e4ed1a3f373a7f6dcccc1 (127.23.214.129:42583)
I20260923 17:23:32.824716 11926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0aedbd58c547ce887c6e18b73f62ed" candidate_uuid: "e8485bea024544e9bac4cb49323afeca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db530a69230e4ed1a3f373a7f6dcccc1"
I20260923 17:23:32.824729 12189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0aedbd58c547ce887c6e18b73f62ed" candidate_uuid: "e8485bea024544e9bac4cb49323afeca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739b1d35bce848b292ce1d89c5b47d89"
I20260923 17:23:32.824823 11926 raft_consensus.cc:3037] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:32.824854 12189 raft_consensus.cc:3037] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:32.825928 12189 raft_consensus.cc:2454] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8485bea024544e9bac4cb49323afeca in term 1.
I20260923 17:23:32.825928 11926 raft_consensus.cc:2454] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8485bea024544e9bac4cb49323afeca in term 1.
I20260923 17:23:32.826242 11992 leader_election.cc:302] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db530a69230e4ed1a3f373a7f6dcccc1, e8485bea024544e9bac4cb49323afeca; no voters:
I20260923 17:23:32.826506 12260 raft_consensus.cc:2781] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:32.826717 12260 raft_consensus.cc:686] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 1 LEADER]: Becoming Leader. State: Replica: e8485bea024544e9bac4cb49323afeca, State: Running, Role: LEADER
I20260923 17:23:32.827060 12260 consensus_queue.cc:237] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } }
I20260923 17:23:32.830336 11783 catalog_manager.cc:5495] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca reported cstate change: term changed from 0 to 1, leader changed from <none> to e8485bea024544e9bac4cb49323afeca (127.23.214.130). New cstate: current_term: 1 leader_uuid: "e8485bea024544e9bac4cb49323afeca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8485bea024544e9bac4cb49323afeca" member_type: VOTER last_known_addr { host: "127.23.214.130" port: 35255 } health_report { overall_health: HEALTHY } } }
I20260923 17:23:32.897519 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.894137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.23.214.130:35255)
0923 17:23:32.894270 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:32.894274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:32.894287 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:23:32.894383 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:32.894385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:32.894441 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:32.894534 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:23:32.894827 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.895314 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.896178 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:32.896388 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:32.896463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:32.896723 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:23:32.897159 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:23:32.897247 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:32.897311 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 17:23:32.897363 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:23:32.903709 11926 raft_consensus.cc:1261] T 2e0aedbd58c547ce887c6e18b73f62ed P db530a69230e4ed1a3f373a7f6dcccc1 [term 1 FOLLOWER]: Refusing update from remote peer e8485bea024544e9bac4cb49323afeca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:32.903714 12189 raft_consensus.cc:1261] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 1 FOLLOWER]: Refusing update from remote peer e8485bea024544e9bac4cb49323afeca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:23:32.904449 12264 consensus_queue.cc:1035] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [LEADER]: Connected to new peer: Peer: permanent_uuid: "739b1d35bce848b292ce1d89c5b47d89" member_type: VOTER last_known_addr { host: "127.23.214.131" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:32.904659 12260 consensus_queue.cc:1035] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [LEADER]: Connected to new peer: Peer: permanent_uuid: "db530a69230e4ed1a3f373a7f6dcccc1" member_type: VOTER last_known_addr { host: "127.23.214.129" port: 42583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:23:32.912864 12268 mvcc.cc:205] Tried to move back new op lower bound from 7332594536045592576 to 7332594535742160896. Current Snapshot: MvccSnapshot[applied={T|T < 7332594536045592576}]
I20260923 17:23:32.921280 11783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:32.921475 11783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:23:32.924084 11783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2d232c55dbc458db721e1a101d0729c: Sending DeleteTablet for 3 replicas of tablet 2e0aedbd58c547ce887c6e18b73f62ed
I20260923 17:23:32.924911 11906 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0aedbd58c547ce887c6e18b73f62ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:32 UTC) from {username='slave'} at 127.0.0.1:57458
I20260923 17:23:32.924957 12168 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0aedbd58c547ce887c6e18b73f62ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:32 UTC) from {username='slave'} at 127.0.0.1:59932
I20260923 17:23:32.925024 12037 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0aedbd58c547ce887c6e18b73f62ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:23:32 UTC) from {username='slave'} at 127.0.0.1:39552
I20260923 17:23:32.925618 12279 tablet_replica.cc:330] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: stopping tablet replica
I20260923 17:23:32.925994 12279 raft_consensus.cc:2229] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:23:32.926192 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11844
I20260923 17:23:32.926445 12279 raft_consensus.cc:2258] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:32.926939 12277 tablet_replica.cc:330] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: stopping tablet replica
I20260923 17:23:32.927233 12277 raft_consensus.cc:2229] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:23:32.927536 12277 raft_consensus.cc:2258] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:23:32.928598 12279 ts_tablet_manager.cc:1905] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:32.929548 12277 ts_tablet_manager.cc:1905] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:23:32.932322 12279 ts_tablet_manager.cc:1918] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:32.932359 12277 ts_tablet_manager.cc:1918] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:23:32.932447 12277 log.cc:1196] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-2/wal/wals/2e0aedbd58c547ce887c6e18b73f62ed
I20260923 17:23:32.932569 12279 log.cc:1196] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Deleting WAL directory at /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790183952943234-24410-0/minicluster-data/ts-1/wal/wals/2e0aedbd58c547ce887c6e18b73f62ed
I20260923 17:23:32.932768 12277 ts_tablet_manager.cc:1939] T 2e0aedbd58c547ce887c6e18b73f62ed P 739b1d35bce848b292ce1d89c5b47d89: Deleting consensus metadata
I20260923 17:23:32.933033 12279 ts_tablet_manager.cc:1939] T 2e0aedbd58c547ce887c6e18b73f62ed P e8485bea024544e9bac4cb49323afeca: Deleting consensus metadata
W20260923 17:23:32.934242 11770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.214.129:42583 (error 108)
I20260923 17:23:32.934290 11769 catalog_manager.cc:4841] TS 739b1d35bce848b292ce1d89c5b47d89 (127.23.214.131:43851): tablet 2e0aedbd58c547ce887c6e18b73f62ed (table test-table [id=5f1a745f437c4163a62e8f7da0797ec0]) successfully deleted
I20260923 17:23:32.934281 11768 catalog_manager.cc:4841] TS e8485bea024544e9bac4cb49323afeca (127.23.214.130:35255): tablet 2e0aedbd58c547ce887c6e18b73f62ed (table test-table [id=5f1a745f437c4163a62e8f7da0797ec0]) successfully deleted
I20260923 17:23:32.935220 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11975
I20260923 17:23:32.936594 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:32.936232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.129:42583 (local address 127.0.0.1:57466)
0923 17:23:32.936400 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:32.936484 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.129:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 17:23:32.936786 11770 catalog_manager.cc:4584] TS db530a69230e4ed1a3f373a7f6dcccc1 (127.23.214.129:42583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e0aedbd58c547ce887c6e18b73f62ed: Network error: Client connection negotiation failed: client connection to 127.23.214.129:42583: connect: Connection refused (error 111)
I20260923 17:23:32.941807 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12106
I20260923 17:23:32.947541 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 11753
2026-09-23T17:23:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1376 ms)
[----------] 6 tests from EncryptionPolicyTest (11267 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:33 dist-test-slave-9q7f krb5kdc[12283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:33 dist-test-slave-9q7f krb5kdc[12283](info): set up 2 sockets
Sep 23 17:23:33 dist-test-slave-9q7f krb5kdc[12283](info): commencing operation
krb5kdc: starting...
W20260923 17:23:34.004153 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:34 dist-test-slave-9q7f krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:34.128981 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:34.241442 12296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:34.241721 12296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:34.241763 12296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:34.245759 12296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:34.245823 12296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:34.245843 12296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:34.245860 12296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:34.245929 12296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:34.251226 12296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:34.252267 12296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:34.253310 12296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:34.262331 12305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:34.262351 12303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:34.262445 12302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:34.262434 12296 server_base.cc:1034] running on GCE node
I20260923 17:23:34.263288 12296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:34.264434 12296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:34.265633 12296 hybrid_clock.cc:648] HybridClock initialized: now 1790184214265606 us; error 50 us; skew 500 ppm
Sep 23 17:23:34 dist-test-slave-9q7f krb5kdc[12283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:34.268589 12296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:34.269595 12296 webserver.cc:460] Webserver started at http://127.0.0.1:44227/ using document root <none> and password file <none>
I20260923 17:23:34.270290 12296 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:34.270398 12296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:34.270641 12296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:34.272850 12296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "500fcd8634d8419b8cd6eef08e31eb81"
format_stamp: "Formatted at 2026-09-23 17:23:34 on dist-test-slave-9q7f"
server_key: "c94dcebee40f49459008a470e598ad33"
server_key_iv: "2e466aa85578fb990828cf5c2e483bd8"
server_key_version: "encryptionkey@0"
I20260923 17:23:34.273506 12296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "500fcd8634d8419b8cd6eef08e31eb81"
format_stamp: "Formatted at 2026-09-23 17:23:34 on dist-test-slave-9q7f"
server_key: "c94dcebee40f49459008a470e598ad33"
server_key_iv: "2e466aa85578fb990828cf5c2e483bd8"
server_key_version: "encryptionkey@0"
I20260923 17:23:34.277470 12296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 17:23:34.280172 12312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:34.281457 12296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:34.281603 12296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "500fcd8634d8419b8cd6eef08e31eb81"
format_stamp: "Formatted at 2026-09-23 17:23:34 on dist-test-slave-9q7f"
server_key: "c94dcebee40f49459008a470e598ad33"
server_key_iv: "2e466aa85578fb990828cf5c2e483bd8"
server_key_version: "encryptionkey@0"
I20260923 17:23:34.281731 12296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:34.302074 12296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:34.305032 12296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:34.305253 12296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:34.312708 12296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45379
I20260923 17:23:34.312731 12364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45379 every 8 connection(s)
I20260923 17:23:34.313714 12296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:34.315768 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12296
I20260923 17:23:34.315912 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:34.316246 24410 external_mini_cluster.cc:1428] Setting key e367e494ce25636fba228e5acfb28719
I20260923 17:23:34.316880 12365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:34.322781 12365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81: Bootstrap starting.
Sep 23 17:23:34 dist-test-slave-9q7f krb5kdc[12283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:23:34.325342 12365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:34.326138 12365 log.cc:824] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:34.328204 12365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81: No bootstrap required, opened a new log
I20260923 17:23:34.330891 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:34.317711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59421 (local address 127.0.0.1:45379)
0923 17:23:34.318176 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:34.318192 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:34.318234 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:23:34.318880 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:34.318895 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:34.319234 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:34.319569 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:34.320417 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:34.321489 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:34.322449 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:34.322957 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:34.323116 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:34.325954 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:34.325960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:34.325974 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:34.326002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:34.327862 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:34.328448 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:34.328453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:34.328457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:34.328565 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:34.328911 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:34.328914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:34.328916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:34.329330 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:34.329552 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:34.329899 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 17:23:34.330128 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":172,"threads_started":1}
I20260923 17:23:34.332231 12365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fcd8634d8419b8cd6eef08e31eb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260923 17:23:34.332484 12365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:34.332561 12365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500fcd8634d8419b8cd6eef08e31eb81, State: Initialized, Role: FOLLOWER
I20260923 17:23:34.333051 12365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fcd8634d8419b8cd6eef08e31eb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260923 17:23:34.333233 12365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:34.333307 12365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:34.333410 12365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:34.334110 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:34.334558 12365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fcd8634d8419b8cd6eef08e31eb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260923 17:23:34.334903 12365 leader_election.cc:302] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500fcd8634d8419b8cd6eef08e31eb81; no voters:
I20260923 17:23:34.335237 12365 leader_election.cc:288] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:34.335390 12370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:34.335588 12370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [term 1 LEADER]: Becoming Leader. State: Replica: 500fcd8634d8419b8cd6eef08e31eb81, State: Running, Role: LEADER
I20260923 17:23:34.335875 12370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fcd8634d8419b8cd6eef08e31eb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260923 17:23:34.336529 12365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:34.337545 12376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500fcd8634d8419b8cd6eef08e31eb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fcd8634d8419b8cd6eef08e31eb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } } }
I20260923 17:23:34.337740 12376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:34.337994 12376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500fcd8634d8419b8cd6eef08e31eb81. Latest consensus state: current_term: 1 leader_uuid: "500fcd8634d8419b8cd6eef08e31eb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500fcd8634d8419b8cd6eef08e31eb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } } }
I20260923 17:23:34.338081 12376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:34.339092 12384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:34.342312 12384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:34.345362 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:34.336190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59467 (local address 127.0.0.1:45379)
0923 17:23:34.336306 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:34.336310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:34.336664 (+ 354us) server_negotiation.cc:407] Connection header received
0923 17:23:34.336845 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:34.336849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:34.336899 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:34.336969 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:34.338239 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:34.338980 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:34.340041 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:34.340292 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:34.340404 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:34.341829 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:34.341833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:34.341836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:34.341860 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:34.343438 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:34.343931 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:34.343934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:34.343936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:34.344022 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:34.344393 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:34.344396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:34.344398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:34.344628 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:34.344787 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:34.345084 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 17:23:34.345190 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:23:34.348224 12384 catalog_manager.cc:1347] Generated new cluster ID: 3155ad8280d54cbe878ed85656c410c9
I20260923 17:23:34.348294 12384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:34.361855 12384 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:34.362876 12384 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:34.367807 12384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500fcd8634d8419b8cd6eef08e31eb81: Generated new TSK 0
I20260923 17:23:34.368569 12384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:34.372704 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12296
W20260923 17:23:34.378373 12281 connection.cc:535] client connection to 127.0.0.1:45379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45379): Connection reset by peer (error 104)
2026-09-23T17:23:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:34 dist-test-slave-9q7f krb5kdc[12396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:34 dist-test-slave-9q7f krb5kdc[12396](info): set up 2 sockets
Sep 23 17:23:34 dist-test-slave-9q7f krb5kdc[12396](info): commencing operation
krb5kdc: starting...
W20260923 17:23:35.404358 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:35 dist-test-slave-9q7f krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:35.527133 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:35.639104 12409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:35.639367 12409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:35.639410 12409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:35.643348 12409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:35.643407 12409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:35.643426 12409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:35.643443 12409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:35.643458 12409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:35.648543 12409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:35.649487 12409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:35.650530 12409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:35.658842 12415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:35.658905 12417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:35.658886 12414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:35.659353 12409 server_base.cc:1034] running on GCE node
I20260923 17:23:35.659974 12409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:35.661302 12409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:35.662506 12409 hybrid_clock.cc:648] HybridClock initialized: now 1790184215662493 us; error 25 us; skew 500 ppm
Sep 23 17:23:35 dist-test-slave-9q7f krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:35.669584 12409 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:35.670619 12409 webserver.cc:460] Webserver started at http://127.0.0.1:34271/ using document root <none> and password file <none>
I20260923 17:23:35.671196 12409 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:35.671289 12409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:35.671514 12409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:35.673558 12409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "3bdda03cd96941e69a843b12e628a714"
format_stamp: "Formatted at 2026-09-23 17:23:35 on dist-test-slave-9q7f"
server_key: "f61b2bb83998db08bcb79ed32fcf7855"
server_key_iv: "11f47de1f45e72e620d7b5dbba85577c"
server_key_version: "encryptionkey@0"
I20260923 17:23:35.674171 12409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "3bdda03cd96941e69a843b12e628a714"
format_stamp: "Formatted at 2026-09-23 17:23:35 on dist-test-slave-9q7f"
server_key: "f61b2bb83998db08bcb79ed32fcf7855"
server_key_iv: "11f47de1f45e72e620d7b5dbba85577c"
server_key_version: "encryptionkey@0"
I20260923 17:23:35.677690 12409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:35.680047 12424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:35.681180 12409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:35.681316 12409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "3bdda03cd96941e69a843b12e628a714"
format_stamp: "Formatted at 2026-09-23 17:23:35 on dist-test-slave-9q7f"
server_key: "f61b2bb83998db08bcb79ed32fcf7855"
server_key_iv: "11f47de1f45e72e620d7b5dbba85577c"
server_key_version: "encryptionkey@0"
I20260923 17:23:35.681439 12409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:35.697341 12409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:35.700285 12409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:35.700484 12409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:35.707755 12409 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37103
I20260923 17:23:35.707782 12477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37103 every 8 connection(s)
I20260923 17:23:35.708772 12409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:35.711512 12478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:35.713528 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12409
I20260923 17:23:35.713673 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:35.714031 24410 external_mini_cluster.cc:1428] Setting key dc31019213b2f122969db4f905e5527f
I20260923 17:23:35.718115 12478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714: Bootstrap starting.
I20260923 17:23:35.720266 12478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:35.721091 12478 log.cc:824] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:35.722879 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:35.715471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33599 (local address 127.0.0.1:37103)
0923 17:23:35.715899 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:35.715913 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:35.715947 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:23:35.716576 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:35.716593 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:35.716959 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:35.717286 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:35.717681 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:35.718740 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:35.719618 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:35.720095 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:35.720267 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:35.721152 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:35.721158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:35.721173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:35.721592 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:35.721644 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:35.721676 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:35.721815 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:35.722034 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 17:23:35.722169 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260923 17:23:35.723086 12478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714: No bootstrap required, opened a new log
I20260923 17:23:35.725767 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:35.726864 12478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdda03cd96941e69a843b12e628a714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37103 } }
I20260923 17:23:35.727123 12478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:35.727178 12478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bdda03cd96941e69a843b12e628a714, State: Initialized, Role: FOLLOWER
I20260923 17:23:35.727618 12478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdda03cd96941e69a843b12e628a714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37103 } }
I20260923 17:23:35.727790 12478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:35.727883 12478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:35.727993 12478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:35.729142 12478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdda03cd96941e69a843b12e628a714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37103 } }
I20260923 17:23:35.729482 12478 leader_election.cc:302] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bdda03cd96941e69a843b12e628a714; no voters:
I20260923 17:23:35.729789 12478 leader_election.cc:288] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:35.729854 12488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:35.730099 12488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [term 1 LEADER]: Becoming Leader. State: Replica: 3bdda03cd96941e69a843b12e628a714, State: Running, Role: LEADER
I20260923 17:23:35.730414 12488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdda03cd96941e69a843b12e628a714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37103 } }
I20260923 17:23:35.731178 12478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:35.732249 12489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bdda03cd96941e69a843b12e628a714. Latest consensus state: current_term: 1 leader_uuid: "3bdda03cd96941e69a843b12e628a714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdda03cd96941e69a843b12e628a714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37103 } } }
I20260923 17:23:35.732388 12489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:35.732517 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:35.727796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40673 (local address 127.0.0.1:37103)
0923 17:23:35.727902 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:35.727906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:35.728138 (+ 232us) server_negotiation.cc:407] Connection header received
0923 17:23:35.728280 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:35.728282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:35.728337 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:35.728406 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:35.728687 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:35.729262 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:35.730260 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:35.730594 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:35.730685 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:35.731760 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:35.731763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:35.731766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:35.731782 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:35.731815 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:35.731831 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:35.731910 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:35.732199 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 17:23:35.732240 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 17:23:35.732265 12490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bdda03cd96941e69a843b12e628a714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdda03cd96941e69a843b12e628a714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37103 } } }
I20260923 17:23:35.732642 12490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714 [sys.catalog]: This master's current role is: LEADER
W20260923 17:23:35.736025 12503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:23:35.736114 12503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:23:35.736193 12497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:35.736846 12497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:35.741765 12497 catalog_manager.cc:1347] Generated new cluster ID: 3dbcbb3d0ca6427fb46fcff863901847
I20260923 17:23:35.741829 12497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:35.753974 12497 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:35.755173 12497 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:35.763424 12497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bdda03cd96941e69a843b12e628a714: Generated new TSK 0
I20260923 17:23:35.764160 12497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:35.772861 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12409
W20260923 17:23:35.778614 12394 connection.cc:535] client connection to 127.0.0.1:37103 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37103): Connection reset by peer (error 104)
2026-09-23T17:23:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:35 dist-test-slave-9q7f krb5kdc[12509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:35 dist-test-slave-9q7f krb5kdc[12509](info): set up 2 sockets
Sep 23 17:23:35 dist-test-slave-9q7f krb5kdc[12509](info): commencing operation
krb5kdc: starting...
W20260923 17:23:36.832213 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:36 dist-test-slave-9q7f krb5kdc[12509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:36.957609 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:37.071316 12522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:37.071579 12522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:37.071621 12522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:37.075634 12522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:37.075695 12522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:37.075716 12522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:37.075732 12522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:37.075748 12522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:37.081017 12522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:37.082131 12522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:37.083122 12522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:37.091814 12530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:37.091830 12527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:37.091894 12528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:37.092160 12522 server_base.cc:1034] running on GCE node
I20260923 17:23:37.092711 12522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:37.093907 12522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:37.095086 12522 hybrid_clock.cc:648] HybridClock initialized: now 1790184217095069 us; error 28 us; skew 500 ppm
Sep 23 17:23:37 dist-test-slave-9q7f krb5kdc[12509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:37.097786 12522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:37.098752 12522 webserver.cc:460] Webserver started at http://127.0.0.1:37129/ using document root <none> and password file <none>
I20260923 17:23:37.099326 12522 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:37.099418 12522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:37.099647 12522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:37.101711 12522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "d1e24cb1bb1249b4a0b7100383ad76d4"
format_stamp: "Formatted at 2026-09-23 17:23:37 on dist-test-slave-9q7f"
server_key: "2befc86a80816a4205dc1ad037e559e4"
server_key_iv: "a1feba90608464b97227ec24e3183d00"
server_key_version: "encryptionkey@0"
I20260923 17:23:37.102335 12522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "d1e24cb1bb1249b4a0b7100383ad76d4"
format_stamp: "Formatted at 2026-09-23 17:23:37 on dist-test-slave-9q7f"
server_key: "2befc86a80816a4205dc1ad037e559e4"
server_key_iv: "a1feba90608464b97227ec24e3183d00"
server_key_version: "encryptionkey@0"
I20260923 17:23:37.106088 12522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:23:37.108599 12537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:37.110062 12522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:37.110209 12522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "d1e24cb1bb1249b4a0b7100383ad76d4"
format_stamp: "Formatted at 2026-09-23 17:23:37 on dist-test-slave-9q7f"
server_key: "2befc86a80816a4205dc1ad037e559e4"
server_key_iv: "a1feba90608464b97227ec24e3183d00"
server_key_version: "encryptionkey@0"
I20260923 17:23:37.110328 12522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:37.115000 12522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:37.117933 12522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:37.118129 12522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:37.125092 12522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41349
I20260923 17:23:37.125119 12590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41349 every 8 connection(s)
I20260923 17:23:37.126088 12522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:37.128839 12591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:37.134086 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12522
I20260923 17:23:37.134233 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:37.134534 24410 external_mini_cluster.cc:1428] Setting key 01c5e240aaab40682ff630fa1dcf73ce
I20260923 17:23:37.134689 12591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4: Bootstrap starting.
I20260923 17:23:37.137557 12591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:37.138494 12591 log.cc:824] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:37.140407 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:37.135950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37227 (local address 127.0.0.1:41349)
0923 17:23:37.136417 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:37.136435 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:37.136479 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:23:37.137333 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:37.137357 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:37.137748 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:37.138162 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:37.138854 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:37.138863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:37.138879 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:37.138891 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:37.138957 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:37.138979 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:37.139066 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:37.139256 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 17:23:37.139454 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260923 17:23:37.140960 12591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4: No bootstrap required, opened a new log
I20260923 17:23:37.143261 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:37.145018 12591 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260923 17:23:37.145342 12591 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:37.145430 12591 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e24cb1bb1249b4a0b7100383ad76d4, State: Initialized, Role: FOLLOWER
I20260923 17:23:37.145991 12591 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260923 17:23:37.146183 12591 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:37.146265 12591 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:37.146374 12591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:37.147154 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:37.145283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40753 (local address 127.0.0.1:41349)
0923 17:23:37.145456 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:37.145460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:37.145649 (+ 189us) server_negotiation.cc:407] Connection header received
0923 17:23:37.145798 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:37.145800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:37.145835 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:37.145979 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:37.146778 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:37.146781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:37.146782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:37.146793 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:37.146820 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:37.146830 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:37.146896 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:37.146999 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 17:23:37.147028 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:23:37.147656 12591 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260923 17:23:37.148036 12591 leader_election.cc:302] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1e24cb1bb1249b4a0b7100383ad76d4; no voters:
I20260923 17:23:37.148372 12591 leader_election.cc:288] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:37.148502 12601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:37.148717 12601 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [term 1 LEADER]: Becoming Leader. State: Replica: d1e24cb1bb1249b4a0b7100383ad76d4, State: Running, Role: LEADER
I20260923 17:23:37.149024 12601 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260923 17:23:37.149614 12591 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:37.150243 12602 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e24cb1bb1249b4a0b7100383ad76d4. Latest consensus state: current_term: 1 leader_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } } }
I20260923 17:23:37.150379 12602 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:37.150684 12603 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e24cb1bb1249b4a0b7100383ad76d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } } }
I20260923 17:23:37.150753 12607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:37.150789 12603 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:37.154001 12607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:37.159780 12607 catalog_manager.cc:1347] Generated new cluster ID: 2df89eff49c64a6092a638810a940c8e
I20260923 17:23:37.159870 12607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:37.174147 12607 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:37.174504 12607 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:37.182626 12607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1e24cb1bb1249b4a0b7100383ad76d4: Generated new TSK 0
I20260923 17:23:37.183348 12607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:37.189098 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12522
2026-09-23T17:23:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:37 dist-test-slave-9q7f krb5kdc[12622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:37 dist-test-slave-9q7f krb5kdc[12622](info): set up 2 sockets
Sep 23 17:23:37 dist-test-slave-9q7f krb5kdc[12622](info): commencing operation
krb5kdc: starting...
W20260923 17:23:38.224124 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:38 dist-test-slave-9q7f krb5kdc[12622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:38.349007 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:38.459759 12635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:38.460106 12635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:38.460183 12635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:38.464702 12635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:38.464792 12635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:38.464825 12635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:38.464864 12635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:38.464915 12635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:38.470496 12635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:38.471596 12635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:38.472613 12635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:38.481176 12640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:38.481176 12643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:38.481278 12635 server_base.cc:1034] running on GCE node
W20260923 17:23:38.481189 12641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:38.482173 12635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:38.483271 12635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:38.484455 12635 hybrid_clock.cc:648] HybridClock initialized: now 1790184218484441 us; error 29 us; skew 500 ppm
Sep 23 17:23:38 dist-test-slave-9q7f krb5kdc[12622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:38.487311 12635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:38.488252 12635 webserver.cc:460] Webserver started at http://127.0.0.1:38615/ using document root <none> and password file <none>
I20260923 17:23:38.488824 12635 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:38.488914 12635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:38.489141 12635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:38.491259 12635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "4f4d141947a143b3b2f9e390beb073e7"
format_stamp: "Formatted at 2026-09-23 17:23:38 on dist-test-slave-9q7f"
server_key: "3d84ae3ba9f3d836726f2b88aa1cfe9d"
server_key_iv: "3c9c44ef0782dae32cdf3dceaa393528"
server_key_version: "encryptionkey@0"
I20260923 17:23:38.491834 12635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "4f4d141947a143b3b2f9e390beb073e7"
format_stamp: "Formatted at 2026-09-23 17:23:38 on dist-test-slave-9q7f"
server_key: "3d84ae3ba9f3d836726f2b88aa1cfe9d"
server_key_iv: "3c9c44ef0782dae32cdf3dceaa393528"
server_key_version: "encryptionkey@0"
I20260923 17:23:38.495911 12635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:23:38.498983 12650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:38.500193 12635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260923 17:23:38.500344 12635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "4f4d141947a143b3b2f9e390beb073e7"
format_stamp: "Formatted at 2026-09-23 17:23:38 on dist-test-slave-9q7f"
server_key: "3d84ae3ba9f3d836726f2b88aa1cfe9d"
server_key_iv: "3c9c44ef0782dae32cdf3dceaa393528"
server_key_version: "encryptionkey@0"
I20260923 17:23:38.500463 12635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:38.521621 12635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:38.524509 12635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:38.524731 12635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:38.531766 12635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44831
I20260923 17:23:38.531841 12703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44831 every 8 connection(s)
I20260923 17:23:38.532745 12635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:38.535354 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12635
I20260923 17:23:38.535511 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:38.535561 12704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:38.535822 24410 external_mini_cluster.cc:1428] Setting key 17ae841183d9f21c584501a28036d4b7
I20260923 17:23:38.541706 12704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7: Bootstrap starting.
I20260923 17:23:38.544080 12704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7: Neither blocks nor log segments found. Creating new log.
Sep 23 17:23:38 dist-test-slave-9q7f krb5kdc[12622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:23:38.544880 12704 log.cc:824] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:38.546777 12704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7: No bootstrap required, opened a new log
I20260923 17:23:38.550011 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:38.537157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36737 (local address 127.0.0.1:44831)
0923 17:23:38.537609 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:38.537623 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:38.537657 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:23:38.538691 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:38.538743 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:38.539085 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:38.539420 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:38.540224 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:38.541188 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:38.542116 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:38.542576 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:38.542754 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:38.545272 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:38.545277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:38.545290 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:38.545316 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:38.547106 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:38.547542 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:38.547547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:38.547551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:38.547668 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:38.547932 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:38.547937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:38.547938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:38.548310 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:38.548502 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:38.548823 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 17:23:38.549176 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":158,"threads_started":1}
I20260923 17:23:38.550644 12704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4d141947a143b3b2f9e390beb073e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260923 17:23:38.550889 12704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:38.550969 12704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4d141947a143b3b2f9e390beb073e7, State: Initialized, Role: FOLLOWER
I20260923 17:23:38.551402 12704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4d141947a143b3b2f9e390beb073e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260923 17:23:38.551556 12704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:38.551638 12704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:38.551733 12704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:38.552775 12704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4d141947a143b3b2f9e390beb073e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260923 17:23:38.553013 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:38.553162 12704 leader_election.cc:302] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f4d141947a143b3b2f9e390beb073e7; no voters:
I20260923 17:23:38.553503 12704 leader_election.cc:288] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:38.553603 12709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:38.553867 12709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [term 1 LEADER]: Becoming Leader. State: Replica: 4f4d141947a143b3b2f9e390beb073e7, State: Running, Role: LEADER
I20260923 17:23:38.554234 12709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4d141947a143b3b2f9e390beb073e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260923 17:23:38.554947 12704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:38.555907 12710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f4d141947a143b3b2f9e390beb073e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4d141947a143b3b2f9e390beb073e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260923 17:23:38.556034 12710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:38.556253 12710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f4d141947a143b3b2f9e390beb073e7. Latest consensus state: current_term: 1 leader_uuid: "4f4d141947a143b3b2f9e390beb073e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4d141947a143b3b2f9e390beb073e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260923 17:23:38.556326 12710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:38.557758 12723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:38.561386 12723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:38.565730 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:38.555250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45299 (local address 127.0.0.1:44831)
0923 17:23:38.555363 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:38.555368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:38.555730 (+ 362us) server_negotiation.cc:407] Connection header received
0923 17:23:38.555938 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:38.555941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:38.555997 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:38.556052 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:38.557070 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:38.558587 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:38.559401 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:38.559644 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:38.559753 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:38.561462 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:38.561470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:38.561476 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:38.561512 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:38.563358 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:38.564002 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:38.564007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:38.564009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:38.564133 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:38.564522 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:38.564527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:38.564528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:38.564797 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:38.564916 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:38.565387 (+ 471us) server_negotiation.cc:299] Negotiation successful
0923 17:23:38.565536 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 17:23:38.568531 12723 catalog_manager.cc:1347] Generated new cluster ID: 18cdf921956f4f2a94239a867f24a25c
I20260923 17:23:38.568608 12723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:38.585218 12723 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:38.586440 12723 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:38.594313 12723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f4d141947a143b3b2f9e390beb073e7: Generated new TSK 0
I20260923 17:23:38.595122 12723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:38.604655 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12635
2026-09-23T17:23:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:38 dist-test-slave-9q7f krb5kdc[12736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:38 dist-test-slave-9q7f krb5kdc[12736](info): set up 2 sockets
Sep 23 17:23:38 dist-test-slave-9q7f krb5kdc[12736](info): commencing operation
krb5kdc: starting...
W20260923 17:23:39.656291 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:39 dist-test-slave-9q7f krb5kdc[12736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:39.777128 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:39.891144 12750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:39.891510 12750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:39.891623 12750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:39.895907 12750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:39.896018 12750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:39.896060 12750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:39.896118 12750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:39.896157 12750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:39.901717 12750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:39.902974 12750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:39.904079 12750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:39.913097 12758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:39.913247 12750 server_base.cc:1034] running on GCE node
W20260923 17:23:39.913131 12756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:39.913141 12755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:39.914022 12750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:39.915154 12750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:39.916342 12750 hybrid_clock.cc:648] HybridClock initialized: now 1790184219916308 us; error 44 us; skew 500 ppm
Sep 23 17:23:39 dist-test-slave-9q7f krb5kdc[12736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:39.919453 12750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:39.920401 12750 webserver.cc:460] Webserver started at http://127.0.0.1:37409/ using document root <none> and password file <none>
I20260923 17:23:39.920938 12750 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:39.921003 12750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:39.921177 12750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:39.923319 12750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "0aa4144b2d6447ee93b9e97fc3207c35"
format_stamp: "Formatted at 2026-09-23 17:23:39 on dist-test-slave-9q7f"
server_key: "08fd76749f9953265998869c4a88f18c"
server_key_iv: "3c11692afaaeed77660acc6e0e83c34e"
server_key_version: "encryptionkey@0"
I20260923 17:23:39.923880 12750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "0aa4144b2d6447ee93b9e97fc3207c35"
format_stamp: "Formatted at 2026-09-23 17:23:39 on dist-test-slave-9q7f"
server_key: "08fd76749f9953265998869c4a88f18c"
server_key_iv: "3c11692afaaeed77660acc6e0e83c34e"
server_key_version: "encryptionkey@0"
I20260923 17:23:39.927743 12750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:23:39.930189 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:39.931480 12750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:23:39.931670 12750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "0aa4144b2d6447ee93b9e97fc3207c35"
format_stamp: "Formatted at 2026-09-23 17:23:39 on dist-test-slave-9q7f"
server_key: "08fd76749f9953265998869c4a88f18c"
server_key_iv: "3c11692afaaeed77660acc6e0e83c34e"
server_key_version: "encryptionkey@0"
I20260923 17:23:39.931787 12750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:39.942152 12750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:39.945536 12750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:39.945770 12750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:39.953454 12750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41855
I20260923 17:23:39.953583 12821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41855 every 8 connection(s)
I20260923 17:23:39.954521 12750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:39.957576 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:39.963440 12822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35: Bootstrap starting.
I20260923 17:23:39.963615 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12750
I20260923 17:23:39.963774 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:39.964082 24410 external_mini_cluster.cc:1428] Setting key 22d75c5eb5b3790c73b2acb660a2dba6
I20260923 17:23:39.966387 12822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:39.967264 12822 log.cc:824] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:39.969280 12822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35: No bootstrap required, opened a new log
I20260923 17:23:39.973217 12822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260923 17:23:39.973476 12822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:39.973557 12822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa4144b2d6447ee93b9e97fc3207c35, State: Initialized, Role: FOLLOWER
I20260923 17:23:39.973729 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:39.965451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59129 (local address 127.0.0.1:41855)
0923 17:23:39.966009 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:39.966028 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:39.966067 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:23:39.966884 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:39.966913 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:39.967282 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:39.967712 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:39.968154 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:39.969162 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:39.970241 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:39.970761 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:39.970943 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:39.971866 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:39.971871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:39.971884 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:39.972287 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:39.972345 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:39.972370 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:39.972479 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:39.972653 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 17:23:39.972873 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":158,"threads_started":1}
I20260923 17:23:39.974113 12822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260923 17:23:39.974287 12822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:39.974360 12822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:39.974467 12822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:39.975695 12822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260923 17:23:39.976150 12822 leader_election.cc:302] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aa4144b2d6447ee93b9e97fc3207c35; no voters:
I20260923 17:23:39.976549 12822 leader_election.cc:288] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:39.976604 12827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:39.976845 12827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [term 1 LEADER]: Becoming Leader. State: Replica: 0aa4144b2d6447ee93b9e97fc3207c35, State: Running, Role: LEADER
I20260923 17:23:39.976893 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:39.977188 12827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } }
I20260923 17:23:39.977998 12822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:39.978678 12828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } } }
I20260923 17:23:39.978803 12828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:39.979156 12829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aa4144b2d6447ee93b9e97fc3207c35. Latest consensus state: current_term: 1 leader_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa4144b2d6447ee93b9e97fc3207c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41855 } } }
I20260923 17:23:39.979211 12839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:39.979281 12829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:39.982556 12839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:39.984985 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:39.979296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35361 (local address 127.0.0.1:41855)
0923 17:23:39.979418 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:39.979423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:39.980015 (+ 592us) server_negotiation.cc:407] Connection header received
0923 17:23:39.980144 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:39.980151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:39.980210 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:39.980301 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:39.980743 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:39.981576 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:39.982920 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:39.983268 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:39.983335 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:39.984386 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:39.984390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:39.984393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:39.984407 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:39.984442 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:39.984454 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:39.984536 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:39.984780 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:23:39.984838 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:23:39.988956 12839 catalog_manager.cc:1347] Generated new cluster ID: 24ce04708fc64dcbb41062afc7195d5c
I20260923 17:23:39.989035 12839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:40.015870 12839 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:40.017228 12839 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:40.023432 12839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aa4144b2d6447ee93b9e97fc3207c35: Generated new TSK 0
I20260923 17:23:40.024403 12839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:40.039855 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12750
2026-09-23T17:23:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:40 dist-test-slave-9q7f krb5kdc[12855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:40 dist-test-slave-9q7f krb5kdc[12855](info): set up 2 sockets
Sep 23 17:23:40 dist-test-slave-9q7f krb5kdc[12855](info): commencing operation
krb5kdc: starting...
W20260923 17:23:41.100142 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:41 dist-test-slave-9q7f krb5kdc[12855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:41.222188 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:41.331750 12868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:41.332063 12868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:41.332137 12868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:41.336143 12868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:41.336237 12868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:41.336277 12868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:41.336315 12868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:41.336350 12868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:41.341602 12868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:41.342741 12868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:41.343808 12868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:41.352416 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:41.352422 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:41.352470 12868 server_base.cc:1034] running on GCE node
W20260923 17:23:41.352422 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:41.353255 12868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:41.354434 12868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:41.355621 12868 hybrid_clock.cc:648] HybridClock initialized: now 1790184221355586 us; error 42 us; skew 500 ppm
Sep 23 17:23:41 dist-test-slave-9q7f krb5kdc[12855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:41.362779 12868 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:41.363730 12868 webserver.cc:460] Webserver started at http://127.0.0.1:38445/ using document root <none> and password file <none>
I20260923 17:23:41.364300 12868 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:41.364364 12868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:41.364588 12868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:41.366722 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "9cf5af9d04464c7ca25db75e7cdc03c1"
format_stamp: "Formatted at 2026-09-23 17:23:41 on dist-test-slave-9q7f"
server_key: "5791007309c9a85d12e37d4cafa7e641"
server_key_iv: "05f948b80169f4177f99422f31229b2e"
server_key_version: "encryptionkey@0"
I20260923 17:23:41.367280 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "9cf5af9d04464c7ca25db75e7cdc03c1"
format_stamp: "Formatted at 2026-09-23 17:23:41 on dist-test-slave-9q7f"
server_key: "5791007309c9a85d12e37d4cafa7e641"
server_key_iv: "05f948b80169f4177f99422f31229b2e"
server_key_version: "encryptionkey@0"
I20260923 17:23:41.370955 12868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:23:41.373301 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:41.374562 12868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:41.374699 12868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "9cf5af9d04464c7ca25db75e7cdc03c1"
format_stamp: "Formatted at 2026-09-23 17:23:41 on dist-test-slave-9q7f"
server_key: "5791007309c9a85d12e37d4cafa7e641"
server_key_iv: "05f948b80169f4177f99422f31229b2e"
server_key_version: "encryptionkey@0"
I20260923 17:23:41.374817 12868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:41.379347 12868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:41.382164 12868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:41.382349 12868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:41.389191 12868 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35477
I20260923 17:23:41.389223 12935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35477 every 8 connection(s)
I20260923 17:23:41.390182 12868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:41.392824 12936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:41.398296 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12868
I20260923 17:23:41.398447 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:41.398439 12936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1: Bootstrap starting.
I20260923 17:23:41.398753 24410 external_mini_cluster.cc:1428] Setting key 7dbb2a5923e3827738c95766858dcc6b
I20260923 17:23:41.401068 12936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:41.401801 12936 log.cc:824] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:41.403977 12936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1: No bootstrap required, opened a new log
I20260923 17:23:41.404101 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:41.400164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52669 (local address 127.0.0.1:35477)
0923 17:23:41.400625 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:41.400640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:41.400682 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:23:41.401409 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:41.401428 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:41.401719 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:41.402067 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:41.402903 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:41.402907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:41.402918 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:41.402928 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:41.402977 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:41.402995 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:41.403082 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:41.403204 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 17:23:41.403387 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260923 17:23:41.406898 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:41.408486 12936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260923 17:23:41.408772 12936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:41.408845 12936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf5af9d04464c7ca25db75e7cdc03c1, State: Initialized, Role: FOLLOWER
I20260923 17:23:41.409312 12936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260923 17:23:41.409555 12936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:41.409646 12936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:41.409795 12936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:41.410775 12936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260923 17:23:41.411129 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:41.409337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40907 (local address 127.0.0.1:35477)
0923 17:23:41.409467 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:41.409471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:41.409668 (+ 197us) server_negotiation.cc:407] Connection header received
0923 17:23:41.409808 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:41.409811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:41.409844 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:41.409951 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:41.410745 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:41.410748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:41.410751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:41.410761 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:41.410786 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:41.410798 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:41.410896 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:41.410970 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 17:23:41.411010 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:23:41.411154 12936 leader_election.cc:302] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf5af9d04464c7ca25db75e7cdc03c1; no voters:
I20260923 17:23:41.411500 12936 leader_election.cc:288] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:41.411581 12946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:41.411784 12946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf5af9d04464c7ca25db75e7cdc03c1, State: Running, Role: LEADER
I20260923 17:23:41.412177 12946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260923 17:23:41.412748 12936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:41.413287 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf5af9d04464c7ca25db75e7cdc03c1. Latest consensus state: current_term: 1 leader_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } } }
I20260923 17:23:41.413439 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:41.414650 12947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf5af9d04464c7ca25db75e7cdc03c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } } }
I20260923 17:23:41.414743 12947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:41.415323 12962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:41.417974 12962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:41.423542 12962 catalog_manager.cc:1347] Generated new cluster ID: c5b8f08f31df4dd1b55427606997d948
I20260923 17:23:41.423623 12962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:41.440116 12962 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:41.440475 12962 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:41.446215 12962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cf5af9d04464c7ca25db75e7cdc03c1: Generated new TSK 0
I20260923 17:23:41.447026 12962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:41.456877 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12868
2026-09-23T17:23:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:41 dist-test-slave-9q7f krb5kdc[12967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:41 dist-test-slave-9q7f krb5kdc[12967](info): set up 2 sockets
Sep 23 17:23:41 dist-test-slave-9q7f krb5kdc[12967](info): commencing operation
krb5kdc: starting...
W20260923 17:23:42.492146 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:42 dist-test-slave-9q7f krb5kdc[12967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:42.614297 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:42.726020 12980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:42.726370 12980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:42.726459 12980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:42.730620 12980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:42.730708 12980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:42.730747 12980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:42.730798 12980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:42.730837 12980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:42.736146 12980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:42.737268 12980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:42.738399 12980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:42.747025 12985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:42.747011 12986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:42.747006 12988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:42.747123 12980 server_base.cc:1034] running on GCE node
I20260923 17:23:42.747884 12980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:42.749063 12980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:42.750254 12980 hybrid_clock.cc:648] HybridClock initialized: now 1790184222750221 us; error 46 us; skew 500 ppm
Sep 23 17:23:42 dist-test-slave-9q7f krb5kdc[12967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:42.753016 12980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:42.753948 12980 webserver.cc:460] Webserver started at http://127.0.0.1:35405/ using document root <none> and password file <none>
I20260923 17:23:42.754516 12980 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:42.754577 12980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:42.754786 12980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:42.756852 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "edf062d125a243958e89deed7bf68cf7"
format_stamp: "Formatted at 2026-09-23 17:23:42 on dist-test-slave-9q7f"
server_key: "adc811923d16ef8d6eb6a112f338ef9e"
server_key_iv: "6664efdc61914d667a1bbc0909e9b942"
server_key_version: "encryptionkey@0"
I20260923 17:23:42.757405 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "edf062d125a243958e89deed7bf68cf7"
format_stamp: "Formatted at 2026-09-23 17:23:42 on dist-test-slave-9q7f"
server_key: "adc811923d16ef8d6eb6a112f338ef9e"
server_key_iv: "6664efdc61914d667a1bbc0909e9b942"
server_key_version: "encryptionkey@0"
I20260923 17:23:42.761204 12980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 17:23:42.763643 12995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:42.764914 12980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:23:42.765038 12980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "edf062d125a243958e89deed7bf68cf7"
format_stamp: "Formatted at 2026-09-23 17:23:42 on dist-test-slave-9q7f"
server_key: "adc811923d16ef8d6eb6a112f338ef9e"
server_key_iv: "6664efdc61914d667a1bbc0909e9b942"
server_key_version: "encryptionkey@0"
I20260923 17:23:42.765142 12980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:42.778620 12980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:42.781821 12980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:42.782095 12980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:42.789311 12980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38547
I20260923 17:23:42.789341 13047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38547 every 8 connection(s)
I20260923 17:23:42.790279 12980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:42.790999 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 12980
I20260923 17:23:42.791137 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:42.791419 24410 external_mini_cluster.cc:1428] Setting key 87e23bb8173cc5a7449c8b38d912c5b4
I20260923 17:23:42.793457 13048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:42.800334 13048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7: Bootstrap starting.
Sep 23 17:23:42 dist-test-slave-9q7f krb5kdc[12967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:23:42.803032 13048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:42.803939 13048 log.cc:824] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:42.805860 13048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7: No bootstrap required, opened a new log
I20260923 17:23:42.806687 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:42.792775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:37985 (local address 127.0.0.1:38547)
0923 17:23:42.793214 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:42.793233 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:42.793276 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:23:42.794137 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:42.794159 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:42.794579 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:42.794942 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:42.795624 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:42.797130 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:42.798012 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:42.798710 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:42.798880 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:42.801639 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:42.801646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:42.801661 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:42.801692 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:42.803707 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:42.804247 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:42.804254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:42.804262 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:42.804403 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:42.804747 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:42.804750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:42.804760 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:42.805131 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:42.805355 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:42.805665 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 17:23:42.805948 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260923 17:23:42.809756 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:42.810088 13048 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf062d125a243958e89deed7bf68cf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 17:23:42.810329 13048 raft_consensus.cc:374] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:42.810415 13048 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf062d125a243958e89deed7bf68cf7, State: Initialized, Role: FOLLOWER
I20260923 17:23:42.810809 13048 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf062d125a243958e89deed7bf68cf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 17:23:42.810959 13048 raft_consensus.cc:388] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:42.811040 13048 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:42.811137 13048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:42.812171 13048 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf062d125a243958e89deed7bf68cf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 17:23:42.812549 13048 leader_election.cc:302] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edf062d125a243958e89deed7bf68cf7; no voters:
I20260923 17:23:42.812897 13048 leader_election.cc:288] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:42.813016 13058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:42.813253 13058 raft_consensus.cc:686] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [term 1 LEADER]: Becoming Leader. State: Replica: edf062d125a243958e89deed7bf68cf7, State: Running, Role: LEADER
I20260923 17:23:42.813581 13058 consensus_queue.cc:237] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf062d125a243958e89deed7bf68cf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 17:23:42.814252 13048 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:42.815253 13060 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edf062d125a243958e89deed7bf68cf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf062d125a243958e89deed7bf68cf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260923 17:23:42.815438 13060 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:42.815814 13059 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf062d125a243958e89deed7bf68cf7. Latest consensus state: current_term: 1 leader_uuid: "edf062d125a243958e89deed7bf68cf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf062d125a243958e89deed7bf68cf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260923 17:23:42.815968 13059 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7 [sys.catalog]: This master's current role is: LEADER
W20260923 17:23:42.819424 13073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:23:42.819515 13073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:23:42.819619 13067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:42.820401 13067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:42.821617 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:42.812063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:53203 (local address 127.0.0.1:38547)
0923 17:23:42.812197 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:42.812202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:42.812391 (+ 189us) server_negotiation.cc:407] Connection header received
0923 17:23:42.812498 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:42.812501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:42.812551 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:42.812641 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:42.813543 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:42.814338 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:42.815591 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:42.815961 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:42.816061 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:42.817811 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:42.817814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:42.817816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:42.817843 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:42.819637 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:42.820215 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:42.820219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:42.820221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:42.820311 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:42.820704 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:42.820708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:42.820710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:42.820932 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:42.821070 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:42.821385 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 17:23:42.821482 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:23:42.826472 13067 catalog_manager.cc:1347] Generated new cluster ID: 2ec2d33d152e4552b9c175f4ca6c013c
I20260923 17:23:42.826538 13067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:42.831537 13067 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:42.832549 13067 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:42.839392 13067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edf062d125a243958e89deed7bf68cf7: Generated new TSK 0
I20260923 17:23:42.840049 13067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:42.850039 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 12980
W20260923 17:23:42.855544 12965 connection.cc:535] client connection to 127.0.0.1:38547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38547): Connection reset by peer (error 104)
2026-09-23T17:23:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:42 dist-test-slave-9q7f krb5kdc[13079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:42 dist-test-slave-9q7f krb5kdc[13079](info): set up 2 sockets
Sep 23 17:23:42 dist-test-slave-9q7f krb5kdc[13079](info): commencing operation
krb5kdc: starting...
W20260923 17:23:43.896162 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:43 dist-test-slave-9q7f krb5kdc[13079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:44.016657 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:44.126332 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:44.126643 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:44.126715 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:44.130889 13092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:44.130980 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:44.131017 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:44.131069 13092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:44.131109 13092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:44.136456 13092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:44.137581 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:44.138702 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:44.147209 13098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:44.147330 13092 server_base.cc:1034] running on GCE node
W20260923 17:23:44.147208 13100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:44.147208 13097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:44.147989 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:44.149114 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:44.150267 13092 hybrid_clock.cc:648] HybridClock initialized: now 1790184224150254 us; error 25 us; skew 500 ppm
Sep 23 17:23:44 dist-test-slave-9q7f krb5kdc[13079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:44.152989 13092 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:44.153968 13092 webserver.cc:460] Webserver started at http://127.0.0.1:34689/ using document root <none> and password file <none>
I20260923 17:23:44.154523 13092 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:44.154583 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:44.154794 13092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:44.156853 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "85dd16d1144048e0aa918090956c88ad"
format_stamp: "Formatted at 2026-09-23 17:23:44 on dist-test-slave-9q7f"
server_key: "599d3e9da40e54b10ba17e38141f220a"
server_key_iv: "82c3ab72de53f894980cb9affd3f1c85"
server_key_version: "encryptionkey@0"
I20260923 17:23:44.157407 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "85dd16d1144048e0aa918090956c88ad"
format_stamp: "Formatted at 2026-09-23 17:23:44 on dist-test-slave-9q7f"
server_key: "599d3e9da40e54b10ba17e38141f220a"
server_key_iv: "82c3ab72de53f894980cb9affd3f1c85"
server_key_version: "encryptionkey@0"
I20260923 17:23:44.161141 13092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 17:23:44.163827 13107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:44.165040 13092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:44.165170 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "85dd16d1144048e0aa918090956c88ad"
format_stamp: "Formatted at 2026-09-23 17:23:44 on dist-test-slave-9q7f"
server_key: "599d3e9da40e54b10ba17e38141f220a"
server_key_iv: "82c3ab72de53f894980cb9affd3f1c85"
server_key_version: "encryptionkey@0"
I20260923 17:23:44.165299 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:44.199347 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:44.202169 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:44.202387 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:44.209275 13092 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38617
I20260923 17:23:44.209302 13159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38617 every 8 connection(s)
I20260923 17:23:44.210211 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:44.212713 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13092
I20260923 17:23:44.212863 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:44.213075 13160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:44.213205 24410 external_mini_cluster.cc:1428] Setting key 73b714b78e247e9b218b54123e350820
I20260923 17:23:44.219285 13160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad: Bootstrap starting.
I20260923 17:23:44.221606 13160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:44.221938 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:44.214605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:39535 (local address 127.0.0.1:38617)
0923 17:23:44.215027 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:44.215041 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:44.215074 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:23:44.215801 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:44.215818 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:44.216143 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:44.216475 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:44.216853 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:44.217777 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:44.218683 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:44.219152 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:44.219345 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:44.220224 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:44.220228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:44.220240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:44.220616 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:44.220668 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:44.220690 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:44.220843 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:44.221017 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 17:23:44.221153 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260923 17:23:44.222520 13160 log.cc:824] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:44.224591 13160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad: No bootstrap required, opened a new log
I20260923 17:23:44.224931 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:44.229271 13160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85dd16d1144048e0aa918090956c88ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38617 } }
I20260923 17:23:44.229547 13160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:44.229628 13160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85dd16d1144048e0aa918090956c88ad, State: Initialized, Role: FOLLOWER
I20260923 17:23:44.230100 13160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85dd16d1144048e0aa918090956c88ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38617 } }
I20260923 17:23:44.230278 13160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:44.230342 13160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:44.230486 13160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:44.231314 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:44.227101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:34041 (local address 127.0.0.1:38617)
0923 17:23:44.227211 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:44.227215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:44.227388 (+ 173us) server_negotiation.cc:407] Connection header received
0923 17:23:44.227500 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:44.227502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:44.227535 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:44.227605 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:44.227898 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:44.228372 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:44.229463 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:44.229787 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:44.229910 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:44.230859 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:44.230862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:44.230865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:44.230877 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:44.230902 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:44.230918 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:44.231010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:44.231136 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 17:23:44.231168 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:23:44.231359 13160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85dd16d1144048e0aa918090956c88ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38617 } }
I20260923 17:23:44.231727 13160 leader_election.cc:302] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85dd16d1144048e0aa918090956c88ad; no voters:
I20260923 17:23:44.232053 13160 leader_election.cc:288] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:44.232168 13170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:44.232381 13170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [term 1 LEADER]: Becoming Leader. State: Replica: 85dd16d1144048e0aa918090956c88ad, State: Running, Role: LEADER
I20260923 17:23:44.232723 13170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85dd16d1144048e0aa918090956c88ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38617 } }
I20260923 17:23:44.233266 13160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:44.233771 13170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85dd16d1144048e0aa918090956c88ad. Latest consensus state: current_term: 1 leader_uuid: "85dd16d1144048e0aa918090956c88ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85dd16d1144048e0aa918090956c88ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38617 } } }
I20260923 17:23:44.233997 13170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:44.234211 13172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85dd16d1144048e0aa918090956c88ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85dd16d1144048e0aa918090956c88ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38617 } } }
I20260923 17:23:44.234313 13172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:44.234407 13178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:44.237473 13178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:44.242506 13178 catalog_manager.cc:1347] Generated new cluster ID: d4c76c699d4b4caba00ff8aa4a11d7f3
I20260923 17:23:44.242573 13178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:44.255406 13178 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:44.256366 13178 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:44.261075 13178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85dd16d1144048e0aa918090956c88ad: Generated new TSK 0
I20260923 17:23:44.261760 13178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:44.270610 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13092
W20260923 17:23:44.275993 13077 connection.cc:535] client connection to 127.0.0.1:38617 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38617): Connection reset by peer (error 104)
2026-09-23T17:23:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:44 dist-test-slave-9q7f krb5kdc[13191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:44 dist-test-slave-9q7f krb5kdc[13191](info): set up 2 sockets
Sep 23 17:23:44 dist-test-slave-9q7f krb5kdc[13191](info): commencing operation
krb5kdc: starting...
W20260923 17:23:45.351992 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:45 dist-test-slave-9q7f krb5kdc[13191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:45.472877 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:45.583088 13204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:45.583402 13204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:45.583499 13204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:45.587608 13204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:45.587694 13204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:45.587733 13204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:45.587787 13204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:45.587821 13204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:45.593071 13204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:45.594189 13204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:45.595209 13204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:45.603658 13210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:45.603658 13209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:45.603659 13212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:45.603819 13204 server_base.cc:1034] running on GCE node
I20260923 17:23:45.604544 13204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:45.605636 13204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:45.606814 13204 hybrid_clock.cc:648] HybridClock initialized: now 1790184225606798 us; error 31 us; skew 500 ppm
Sep 23 17:23:45 dist-test-slave-9q7f krb5kdc[13191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:45.609653 13204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:45.610575 13204 webserver.cc:460] Webserver started at http://127.0.0.1:40045/ using document root <none> and password file <none>
I20260923 17:23:45.611145 13204 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:45.611205 13204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:45.611370 13204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:45.613370 13204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "2d53747a31374d969730e450a4dad225"
format_stamp: "Formatted at 2026-09-23 17:23:45 on dist-test-slave-9q7f"
server_key: "b05151dea53fe38193c11ff70e15af3e"
server_key_iv: "d42c2997bbcfca16704ac3e8d3885e7a"
server_key_version: "encryptionkey@0"
I20260923 17:23:45.613952 13204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "2d53747a31374d969730e450a4dad225"
format_stamp: "Formatted at 2026-09-23 17:23:45 on dist-test-slave-9q7f"
server_key: "b05151dea53fe38193c11ff70e15af3e"
server_key_iv: "d42c2997bbcfca16704ac3e8d3885e7a"
server_key_version: "encryptionkey@0"
I20260923 17:23:45.617621 13204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:23:45.620039 13219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:45.621254 13204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:45.621379 13204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "2d53747a31374d969730e450a4dad225"
format_stamp: "Formatted at 2026-09-23 17:23:45 on dist-test-slave-9q7f"
server_key: "b05151dea53fe38193c11ff70e15af3e"
server_key_iv: "d42c2997bbcfca16704ac3e8d3885e7a"
server_key_version: "encryptionkey@0"
I20260923 17:23:45.621482 13204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:45.626050 13204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:45.629329 13204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:45.629531 13204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:45.636325 13204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39775
I20260923 17:23:45.636394 13271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39775 every 8 connection(s)
I20260923 17:23:45.637290 13204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:45.638872 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13204
I20260923 17:23:45.639035 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:45.639341 24410 external_mini_cluster.cc:1428] Setting key 9a7b7bf48f15c9abb9eb35dd243f8514
I20260923 17:23:45.640321 13272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:45.644610 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:45.640741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:46741 (local address 127.0.0.1:39775)
0923 17:23:45.641173 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:45.641188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:45.641224 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:23:45.641866 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:45.641937 (+ 71us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:45.642235 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:45.642579 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:45.643426 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:45.643431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:45.643442 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:45.643452 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:45.643509 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:45.643528 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:45.643677 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:45.643746 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 17:23:45.643915 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":125,"threads_started":1}
I20260923 17:23:45.646523 13272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225: Bootstrap starting.
I20260923 17:23:45.647507 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:45.648972 13272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:45.649856 13272 log.cc:824] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:45.651469 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:45.649670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:40603 (local address 127.0.0.1:39775)
0923 17:23:45.649782 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:45.649786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:45.650025 (+ 239us) server_negotiation.cc:407] Connection header received
0923 17:23:45.650129 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:45.650131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:45.650163 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:45.650228 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:45.651048 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:45.651051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:45.651053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:45.651063 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:45.651088 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:45.651098 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:45.651165 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:45.651280 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 17:23:45.651315 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 17:23:45.652215 13272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225: No bootstrap required, opened a new log
I20260923 17:23:45.655788 13272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d53747a31374d969730e450a4dad225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260923 17:23:45.656047 13272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:45.656123 13272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d53747a31374d969730e450a4dad225, State: Initialized, Role: FOLLOWER
I20260923 17:23:45.656589 13272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d53747a31374d969730e450a4dad225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260923 17:23:45.656737 13272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:45.656819 13272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:45.656921 13272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:45.657846 13272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d53747a31374d969730e450a4dad225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260923 17:23:45.658209 13272 leader_election.cc:302] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d53747a31374d969730e450a4dad225; no voters:
I20260923 17:23:45.658517 13272 leader_election.cc:288] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:45.658612 13282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:45.658871 13282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [term 1 LEADER]: Becoming Leader. State: Replica: 2d53747a31374d969730e450a4dad225, State: Running, Role: LEADER
I20260923 17:23:45.659199 13282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d53747a31374d969730e450a4dad225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260923 17:23:45.659816 13272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:45.660862 13284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d53747a31374d969730e450a4dad225. Latest consensus state: current_term: 1 leader_uuid: "2d53747a31374d969730e450a4dad225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d53747a31374d969730e450a4dad225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } } }
I20260923 17:23:45.660986 13284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:45.661478 13283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d53747a31374d969730e450a4dad225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d53747a31374d969730e450a4dad225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } } }
I20260923 17:23:45.661646 13283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:45.662173 13291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:45.664722 13291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:45.670166 13291 catalog_manager.cc:1347] Generated new cluster ID: e25d7d8b2814419ea15ce3fd4d36db71
I20260923 17:23:45.670230 13291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:45.684005 13291 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:45.684307 13291 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:45.692211 13291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d53747a31374d969730e450a4dad225: Generated new TSK 0
I20260923 17:23:45.692963 13291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:45.696777 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13204
2026-09-23T17:23:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:45 dist-test-slave-9q7f krb5kdc[13303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:45 dist-test-slave-9q7f krb5kdc[13303](info): set up 2 sockets
Sep 23 17:23:45 dist-test-slave-9q7f krb5kdc[13303](info): commencing operation
krb5kdc: starting...
W20260923 17:23:46.736150 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:46 dist-test-slave-9q7f krb5kdc[13303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:46.861765 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:46.975278 13316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:46.975596 13316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:46.975677 13316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:46.979696 13316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:46.979801 13316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:46.979849 13316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:46.979908 13316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:46.979947 13316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:46.985455 13316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:46.986565 13316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:46.987597 13316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:46.996153 13324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:46.996178 13316 server_base.cc:1034] running on GCE node
W20260923 17:23:46.996130 13321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:46.996130 13322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:46.997027 13316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:46.998234 13316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:46.999434 13316 hybrid_clock.cc:648] HybridClock initialized: now 1790184226999405 us; error 42 us; skew 500 ppm
Sep 23 17:23:47 dist-test-slave-9q7f krb5kdc[13303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:47.010406 13316 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:47.011389 13316 webserver.cc:460] Webserver started at http://127.0.0.1:35463/ using document root <none> and password file <none>
I20260923 17:23:47.011996 13316 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:47.012091 13316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:47.012333 13316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:47.014556 13316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "1375b2c919504cbeb2774bce5af26a03"
format_stamp: "Formatted at 2026-09-23 17:23:47 on dist-test-slave-9q7f"
server_key: "3822f0ea0b8c4f26b1d3ec8df61f40b4"
server_key_iv: "3132703ed4b116d1d5f3b0d2c622f136"
server_key_version: "encryptionkey@0"
I20260923 17:23:47.015148 13316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "1375b2c919504cbeb2774bce5af26a03"
format_stamp: "Formatted at 2026-09-23 17:23:47 on dist-test-slave-9q7f"
server_key: "3822f0ea0b8c4f26b1d3ec8df61f40b4"
server_key_iv: "3132703ed4b116d1d5f3b0d2c622f136"
server_key_version: "encryptionkey@0"
I20260923 17:23:47.019047 13316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:23:47.021571 13331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:47.023212 13316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:23:47.023357 13316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "1375b2c919504cbeb2774bce5af26a03"
format_stamp: "Formatted at 2026-09-23 17:23:47 on dist-test-slave-9q7f"
server_key: "3822f0ea0b8c4f26b1d3ec8df61f40b4"
server_key_iv: "3132703ed4b116d1d5f3b0d2c622f136"
server_key_version: "encryptionkey@0"
I20260923 17:23:47.023476 13316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:47.046490 13316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:47.049980 13316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:47.050186 13316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:47.057214 13316 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35091
I20260923 17:23:47.057291 13383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35091 every 8 connection(s)
I20260923 17:23:47.058259 13316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:47.060998 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:47.066414 13384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03: Bootstrap starting.
I20260923 17:23:47.068389 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13316
I20260923 17:23:47.068522 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:47.068833 24410 external_mini_cluster.cc:1428] Setting key 1208dac021a6650c9bf9c6a7dc356a9e
I20260923 17:23:47.069061 13384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:47.070016 13384 log.cc:824] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:47.072243 13384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03: No bootstrap required, opened a new log
I20260923 17:23:47.076611 13384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1375b2c919504cbeb2774bce5af26a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260923 17:23:47.076843 13384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:47.076879 13384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1375b2c919504cbeb2774bce5af26a03, State: Initialized, Role: FOLLOWER
I20260923 17:23:47.077360 13384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1375b2c919504cbeb2774bce5af26a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260923 17:23:47.077484 13384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:47.077574 13384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:47.077708 13384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 17:23:47 dist-test-slave-9q7f krb5kdc[13303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:23:47.078819 13384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1375b2c919504cbeb2774bce5af26a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260923 17:23:47.079221 13384 leader_election.cc:302] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1375b2c919504cbeb2774bce5af26a03; no voters:
I20260923 17:23:47.079632 13384 leader_election.cc:288] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:47.079784 13389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:47.080050 13389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [term 1 LEADER]: Becoming Leader. State: Replica: 1375b2c919504cbeb2774bce5af26a03, State: Running, Role: LEADER
I20260923 17:23:47.080433 13389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1375b2c919504cbeb2774bce5af26a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260923 17:23:47.081051 13384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:47.082260 13391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1375b2c919504cbeb2774bce5af26a03. Latest consensus state: current_term: 1 leader_uuid: "1375b2c919504cbeb2774bce5af26a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1375b2c919504cbeb2774bce5af26a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } } }
I20260923 17:23:47.082391 13391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:47.082616 13390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1375b2c919504cbeb2774bce5af26a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1375b2c919504cbeb2774bce5af26a03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } } }
I20260923 17:23:47.082707 13390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:47.083034 13398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:47.085471 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.069957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:38589 (local address 127.0.0.1:35091)
0923 17:23:47.070419 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.070438 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.070480 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:23:47.071286 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.071309 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.071726 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.072102 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:47.072858 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.074360 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.075239 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.075988 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.076156 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.078999 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:47.079006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:47.079023 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:47.079058 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:47.081379 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.082110 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.082119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.082129 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.082272 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.082699 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.082703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.082705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.083280 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:47.083507 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.084300 (+ 793us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.084541 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
I20260923 17:23:47.086721 13398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:47.088704 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:47.092168 13398 catalog_manager.cc:1347] Generated new cluster ID: cc363be1410b40998879588aec73a97b
I20260923 17:23:47.092226 13398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:47.100518 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:47.090954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:50561 (local address 127.0.0.1:35091)
0923 17:23:47.091045 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:47.091049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:47.091308 (+ 259us) server_negotiation.cc:407] Connection header received
0923 17:23:47.091411 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:47.091413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:47.091459 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:47.091543 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:47.092458 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.093296 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.094528 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:47.094892 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:47.094970 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:47.096805 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:47.096809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:47.096812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:47.096840 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:47.098451 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.099036 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.099040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.099042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.099125 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:47.099502 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:47.099505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:47.099507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:47.099708 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:47.099835 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:47.100254 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 17:23:47.100353 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 17:23:47.123160 13398 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:47.124185 13398 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:47.140617 13398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1375b2c919504cbeb2774bce5af26a03: Generated new TSK 0
I20260923 17:23:47.141353 13398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:47.147881 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13316
W20260923 17:23:47.153470 13301 connection.cc:535] client connection to 127.0.0.1:35091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35091): Connection reset by peer (error 104)
2026-09-23T17:23:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:47 dist-test-slave-9q7f krb5kdc[13415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:47 dist-test-slave-9q7f krb5kdc[13415](info): set up 2 sockets
Sep 23 17:23:47 dist-test-slave-9q7f krb5kdc[13415](info): commencing operation
krb5kdc: starting...
W20260923 17:23:48.188382 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:48 dist-test-slave-9q7f krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:48.308645 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:48.421975 13428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:48.422232 13428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:48.422273 13428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:48.426205 13428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:48.426255 13428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:48.426275 13428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:48.426291 13428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:48.426307 13428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:48.431463 13428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:48.432458 13428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:48.433422 13428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:48.441675 13434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:48.441951 13436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:48.441725 13433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:48.442019 13428 server_base.cc:1034] running on GCE node
I20260923 17:23:48.442593 13428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:48.443687 13428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:48.444881 13428 hybrid_clock.cc:648] HybridClock initialized: now 1790184228444849 us; error 42 us; skew 500 ppm
Sep 23 17:23:48 dist-test-slave-9q7f krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:48.447613 13428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:48.448487 13428 webserver.cc:460] Webserver started at http://127.0.0.1:42637/ using document root <none> and password file <none>
I20260923 17:23:48.449043 13428 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:48.449134 13428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:48.449374 13428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:48.451453 13428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "8341c1327c1443cc80d07ad8cc38734c"
format_stamp: "Formatted at 2026-09-23 17:23:48 on dist-test-slave-9q7f"
server_key: "d6dcdcbff4f21b02a3a18cb48af4bf82"
server_key_iv: "122965e512ea6746d2f757c67cb9f93a"
server_key_version: "encryptionkey@0"
I20260923 17:23:48.452003 13428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "8341c1327c1443cc80d07ad8cc38734c"
format_stamp: "Formatted at 2026-09-23 17:23:48 on dist-test-slave-9q7f"
server_key: "d6dcdcbff4f21b02a3a18cb48af4bf82"
server_key_iv: "122965e512ea6746d2f757c67cb9f93a"
server_key_version: "encryptionkey@0"
I20260923 17:23:48.455652 13428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 17:23:48.458043 13443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:48.459242 13428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:48.459383 13428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "8341c1327c1443cc80d07ad8cc38734c"
format_stamp: "Formatted at 2026-09-23 17:23:48 on dist-test-slave-9q7f"
server_key: "d6dcdcbff4f21b02a3a18cb48af4bf82"
server_key_iv: "122965e512ea6746d2f757c67cb9f93a"
server_key_version: "encryptionkey@0"
I20260923 17:23:48.459496 13428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:48.472544 13428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:48.475401 13428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:48.475610 13428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:48.482810 13428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34203
I20260923 17:23:48.482836 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34203 every 8 connection(s)
I20260923 17:23:48.483747 13428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:48.484620 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13428
I20260923 17:23:48.484838 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:48.485158 24410 external_mini_cluster.cc:1428] Setting key fcf6f695ded83128898ba69ea0de95a8
I20260923 17:23:48.487287 13496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:48.494392 13496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c: Bootstrap starting.
I20260923 17:23:48.494781 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:48.486595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:36027 (local address 127.0.0.1:34203)
0923 17:23:48.487013 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:48.487034 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:48.487081 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:23:48.487961 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:48.487980 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:48.488327 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:48.488607 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:48.489030 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:48.490116 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:48.491461 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:48.491927 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:48.492042 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:48.493060 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:48.493065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:48.493076 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:48.493445 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:48.493497 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:48.493518 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:48.493613 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:48.493838 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 17:23:48.494012 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":145,"threads_started":1}
I20260923 17:23:48.496975 13496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:48.497910 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:48.497975 13496 log.cc:824] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:48.500315 13496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c: No bootstrap required, opened a new log
I20260923 17:23:48.504566 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:48.500176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:41571 (local address 127.0.0.1:34203)
0923 17:23:48.500297 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:48.500300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:48.500457 (+ 157us) server_negotiation.cc:407] Connection header received
0923 17:23:48.500625 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:48.500628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:48.500667 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:48.500737 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:48.501037 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:48.501516 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:48.502872 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:48.503082 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:48.503157 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:48.504019 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:48.504022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:48.504024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:48.504034 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:48.504056 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:48.504065 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:48.504133 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:48.504337 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 17:23:48.504409 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:23:48.505383 13496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8341c1327c1443cc80d07ad8cc38734c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260923 17:23:48.505690 13496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:48.505756 13496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8341c1327c1443cc80d07ad8cc38734c, State: Initialized, Role: FOLLOWER
I20260923 17:23:48.506225 13496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8341c1327c1443cc80d07ad8cc38734c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260923 17:23:48.506388 13496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:48.506461 13496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:48.506565 13496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:48.507551 13496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8341c1327c1443cc80d07ad8cc38734c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260923 17:23:48.507916 13496 leader_election.cc:302] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8341c1327c1443cc80d07ad8cc38734c; no voters:
I20260923 17:23:48.508287 13496 leader_election.cc:288] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:48.508574 13506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:48.508862 13506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [term 1 LEADER]: Becoming Leader. State: Replica: 8341c1327c1443cc80d07ad8cc38734c, State: Running, Role: LEADER
I20260923 17:23:48.509155 13506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8341c1327c1443cc80d07ad8cc38734c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260923 17:23:48.509682 13496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:48.510959 13508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8341c1327c1443cc80d07ad8cc38734c. Latest consensus state: current_term: 1 leader_uuid: "8341c1327c1443cc80d07ad8cc38734c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8341c1327c1443cc80d07ad8cc38734c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260923 17:23:48.510953 13507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8341c1327c1443cc80d07ad8cc38734c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8341c1327c1443cc80d07ad8cc38734c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260923 17:23:48.511094 13508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:48.511094 13507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:48.511512 13515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:48.514425 13515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:48.519964 13515 catalog_manager.cc:1347] Generated new cluster ID: 7b1d6e39218a4795b3fa25c132eeda46
I20260923 17:23:48.520040 13515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:48.532882 13515 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:48.533831 13515 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:48.543049 13515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8341c1327c1443cc80d07ad8cc38734c: Generated new TSK 0
I20260923 17:23:48.543787 13515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:48.547761 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13428
W20260923 17:23:48.552786 13413 connection.cc:535] client connection to 127.0.0.1:34203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34203): Connection reset by peer (error 104)
2026-09-23T17:23:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:48 dist-test-slave-9q7f krb5kdc[13527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:48 dist-test-slave-9q7f krb5kdc[13527](info): set up 2 sockets
Sep 23 17:23:48 dist-test-slave-9q7f krb5kdc[13527](info): commencing operation
krb5kdc: starting...
W20260923 17:23:49.660120 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:49 dist-test-slave-9q7f krb5kdc[13527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:23:49.782521 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:49.894325 13540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:49.894598 13540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:49.894640 13540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:49.898636 13540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:49.898694 13540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:49.898713 13540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:49.898730 13540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:49.898746 13540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:49.903954 13540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:49.904969 13540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:49.906055 13540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:49.914438 13548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:49.914438 13546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:49.914450 13545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:49.914752 13540 server_base.cc:1034] running on GCE node
I20260923 17:23:49.915362 13540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:49.916549 13540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:49.917738 13540 hybrid_clock.cc:648] HybridClock initialized: now 1790184229917708 us; error 53 us; skew 500 ppm
Sep 23 17:23:49 dist-test-slave-9q7f krb5kdc[13527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:49.920482 13540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:23:49.921442 13540 webserver.cc:460] Webserver started at http://127.0.0.1:33905/ using document root <none> and password file <none>
I20260923 17:23:49.922091 13540 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:49.922185 13540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:49.922420 13540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:49.924484 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "c30a523ccbab433c9a40f5cf0e44fa2c"
format_stamp: "Formatted at 2026-09-23 17:23:49 on dist-test-slave-9q7f"
server_key: "b1ce9c4122acbd433737cb2705af6236"
server_key_iv: "6035dc82bc63c8851a19ddbf8f0ef1dc"
server_key_version: "encryptionkey@0"
I20260923 17:23:49.925047 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "c30a523ccbab433c9a40f5cf0e44fa2c"
format_stamp: "Formatted at 2026-09-23 17:23:49 on dist-test-slave-9q7f"
server_key: "b1ce9c4122acbd433737cb2705af6236"
server_key_iv: "6035dc82bc63c8851a19ddbf8f0ef1dc"
server_key_version: "encryptionkey@0"
I20260923 17:23:49.928687 13540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 17:23:49.931191 13555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:49.932442 13540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:49.932593 13540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "c30a523ccbab433c9a40f5cf0e44fa2c"
format_stamp: "Formatted at 2026-09-23 17:23:49 on dist-test-slave-9q7f"
server_key: "b1ce9c4122acbd433737cb2705af6236"
server_key_iv: "6035dc82bc63c8851a19ddbf8f0ef1dc"
server_key_version: "encryptionkey@0"
I20260923 17:23:49.932708 13540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:49.937726 13540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:49.940711 13540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:49.940948 13540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:49.948387 13540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42963
I20260923 17:23:49.948400 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42963 every 8 connection(s)
I20260923 17:23:49.949378 13540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:49.952239 13608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:49.957701 13608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c: Bootstrap starting.
I20260923 17:23:49.959048 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13540
I20260923 17:23:49.959180 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:49.959522 24410 external_mini_cluster.cc:1428] Setting key 9be4b66b088697691d1de10d2f85481c
I20260923 17:23:49.960899 13608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:49.961695 13608 log.cc:824] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:49.963867 13608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c: No bootstrap required, opened a new log
I20260923 17:23:49.964659 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:49.960643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:43857 (local address 127.0.0.1:42963)
0923 17:23:49.961082 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:49.961098 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:49.961134 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:23:49.961781 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:49.961797 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:49.962190 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:49.962546 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:49.963357 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:49.963362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:49.963375 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:49.963386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:49.963440 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:49.963459 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:49.963574 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:49.963688 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 17:23:49.963865 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":149,"threads_started":1}
I20260923 17:23:49.967617 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:49.968247 13608 raft_consensus.cc:348] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260923 17:23:49.968494 13608 raft_consensus.cc:374] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:49.968572 13608 raft_consensus.cc:729] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30a523ccbab433c9a40f5cf0e44fa2c, State: Initialized, Role: FOLLOWER
I20260923 17:23:49.969076 13608 consensus_queue.cc:260] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260923 17:23:49.969254 13608 raft_consensus.cc:388] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:49.969344 13608 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:49.969442 13608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:49.970809 13608 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260923 17:23:49.971216 13608 leader_election.cc:302] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c30a523ccbab433c9a40f5cf0e44fa2c; no voters:
I20260923 17:23:49.971405 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:49.969831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:38161 (local address 127.0.0.1:42963)
0923 17:23:49.969973 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:49.969978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:49.970120 (+ 142us) server_negotiation.cc:407] Connection header received
0923 17:23:49.970202 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:49.970204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:49.970241 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:49.970327 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:49.971035 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:49.971038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:49.971040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:49.971052 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:49.971081 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:49.971094 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:49.971181 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:49.971241 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 17:23:49.971279 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:23:49.971611 13608 leader_election.cc:288] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:49.971704 13618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:49.971935 13618 raft_consensus.cc:686] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [term 1 LEADER]: Becoming Leader. State: Replica: c30a523ccbab433c9a40f5cf0e44fa2c, State: Running, Role: LEADER
I20260923 17:23:49.972282 13618 consensus_queue.cc:237] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260923 17:23:49.972893 13608 sys_catalog.cc:564] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:49.974125 13619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260923 17:23:49.974112 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30a523ccbab433c9a40f5cf0e44fa2c. Latest consensus state: current_term: 1 leader_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30a523ccbab433c9a40f5cf0e44fa2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260923 17:23:49.974272 13619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:49.974277 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:49.974761 13627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:49.978031 13627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:49.984105 13627 catalog_manager.cc:1347] Generated new cluster ID: 65665a378d784f2f92bfa1d716189d23
I20260923 17:23:49.984184 13627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:50.005170 13627 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:50.005518 13627 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:50.009145 13627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c30a523ccbab433c9a40f5cf0e44fa2c: Generated new TSK 0
I20260923 17:23:50.009779 13627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:50.016512 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13540
2026-09-23T17:23:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:50 dist-test-slave-9q7f krb5kdc[13639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:50 dist-test-slave-9q7f krb5kdc[13639](info): set up 2 sockets
Sep 23 17:23:50 dist-test-slave-9q7f krb5kdc[13639](info): commencing operation
krb5kdc: starting...
W20260923 17:23:51.080075 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:51 dist-test-slave-9q7f krb5kdc[13639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:51.206068 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:33645
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:33645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:51.315891 13652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:51.316166 13652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:51.316210 13652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:51.320266 13652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:51.320341 13652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:51.320361 13652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:51.320379 13652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:51.320393 13652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:51.325670 13652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:33645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:33645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:51.326812 13652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:51.327788 13652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:51.336470 13660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:51.336488 13658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:51.336481 13657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:51.336571 13652 server_base.cc:1034] running on GCE node
I20260923 17:23:51.337247 13652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:51.338439 13652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:51.339632 13652 hybrid_clock.cc:648] HybridClock initialized: now 1790184231339617 us; error 26 us; skew 500 ppm
Sep 23 17:23:51 dist-test-slave-9q7f krb5kdc[13639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184231, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:51.342502 13652 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:51.343350 13652 webserver.cc:460] Webserver started at http://127.23.214.190:41885/ using document root <none> and password file <none>
I20260923 17:23:51.343905 13652 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:51.343999 13652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:51.344228 13652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:51.346326 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "b5dd113183bf47d4ab002e9741e3f6b5"
format_stamp: "Formatted at 2026-09-23 17:23:51 on dist-test-slave-9q7f"
server_key: "5d304de8d019d371fe8113a946a54611"
server_key_iv: "3fe9368e52ecbdaf4d2e9778444efbd3"
server_key_version: "encryptionkey@0"
I20260923 17:23:51.346918 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "b5dd113183bf47d4ab002e9741e3f6b5"
format_stamp: "Formatted at 2026-09-23 17:23:51 on dist-test-slave-9q7f"
server_key: "5d304de8d019d371fe8113a946a54611"
server_key_iv: "3fe9368e52ecbdaf4d2e9778444efbd3"
server_key_version: "encryptionkey@0"
I20260923 17:23:51.350539 13652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:51.352945 13667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:51.354269 13652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:51.354439 13652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "b5dd113183bf47d4ab002e9741e3f6b5"
format_stamp: "Formatted at 2026-09-23 17:23:51 on dist-test-slave-9q7f"
server_key: "5d304de8d019d371fe8113a946a54611"
server_key_iv: "3fe9368e52ecbdaf4d2e9778444efbd3"
server_key_version: "encryptionkey@0"
I20260923 17:23:51.354588 13652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:51.370101 13652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:51.373816 13652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:51.374084 13652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:51.381453 13652 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:33645
I20260923 17:23:51.381470 13719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:33645 every 8 connection(s)
I20260923 17:23:51.382572 13652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:51.385380 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:51.390924 13720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5: Bootstrap starting.
I20260923 17:23:51.392690 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13652
I20260923 17:23:51.392807 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:51.393108 24410 external_mini_cluster.cc:1428] Setting key 771a67c2fa33f95bd4ab39836c8f6c3b
I20260923 17:23:51.393457 13720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:51.394513 13720 log.cc:824] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:51.396564 13720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5: No bootstrap required, opened a new log
I20260923 17:23:51.400789 13720 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33645 } }
I20260923 17:23:51.401090 13720 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:51.401165 13720 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dd113183bf47d4ab002e9741e3f6b5, State: Initialized, Role: FOLLOWER
I20260923 17:23:51.401691 13720 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33645 } }
I20260923 17:23:51.401913 13720 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:51.401990 13720 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:51.402091 13720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 17:23:51 dist-test-slave-9q7f krb5kdc[13639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:51.403105 13720 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33645 } }
I20260923 17:23:51.403560 13720 leader_election.cc:302] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5dd113183bf47d4ab002e9741e3f6b5; no voters:
I20260923 17:23:51.403955 13720 leader_election.cc:288] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:51.404030 13725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:51.404245 13725 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [term 1 LEADER]: Becoming Leader. State: Replica: b5dd113183bf47d4ab002e9741e3f6b5, State: Running, Role: LEADER
I20260923 17:23:51.404594 13725 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33645 } }
I20260923 17:23:51.405362 13720 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:51.405923 13726 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5dd113183bf47d4ab002e9741e3f6b5. Latest consensus state: current_term: 1 leader_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33645 } } }
I20260923 17:23:51.406060 13726 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:51.406358 13727 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dd113183bf47d4ab002e9741e3f6b5" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 33645 } } }
I20260923 17:23:51.406495 13727 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:51.409369 13723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:51.394308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36541 (local address 127.23.214.190:33645)
0923 17:23:51.394788 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:51.394805 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:51.394847 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:23:51.395658 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:51.395678 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:51.396016 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:51.396286 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:51.397211 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:51.398231 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:51.399511 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:51.399984 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:51.400098 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:51.403072 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:51.403077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:51.403091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:51.403118 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:51.405230 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:51.405826 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:51.405832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:51.405838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:51.406016 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:51.406504 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:51.406511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:51.406513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:51.407035 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:51.407289 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:51.408213 (+ 924us) server_negotiation.cc:299] Negotiation successful
0923 17:23:51.408439 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":157,"threads_started":1}
W20260923 17:23:51.410758 13740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:23:51.410872 13740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:23:51.410976 13730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:51.411851 13730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:51.412707 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:51.417964 13730 catalog_manager.cc:1347] Generated new cluster ID: d3bf408f3b204590bea296fb13e6b300
I20260923 17:23:51.418028 13730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:51.423434 13723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:51.414898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50753 (local address 127.23.214.190:33645)
0923 17:23:51.415005 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:51.415009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:51.415187 (+ 178us) server_negotiation.cc:407] Connection header received
0923 17:23:51.415304 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:51.415307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:51.415349 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:51.415442 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:51.416168 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:51.416969 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:51.418106 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:51.418353 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:51.418410 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:51.419918 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:51.419921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:51.419924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:51.419948 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:51.421488 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:51.422071 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:51.422074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:51.422076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:51.422144 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:51.422479 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:51.422483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:51.422484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:51.422651 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:51.422808 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:51.423073 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 17:23:51.423166 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 17:23:51.441586 13730 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:51.442894 13730 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:51.452579 13730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5dd113183bf47d4ab002e9741e3f6b5: Generated new TSK 0
I20260923 17:23:51.453416 13730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:51.465530 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13652
W20260923 17:23:51.471323 13637 connection.cc:535] client connection to 127.23.214.190:33645 recv error: Network error: failed to read from TLS socket (remote: 127.23.214.190:33645): Connection reset by peer (error 104)
2026-09-23T17:23:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:51 dist-test-slave-9q7f krb5kdc[13751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:51 dist-test-slave-9q7f krb5kdc[13751](info): set up 2 sockets
Sep 23 17:23:51 dist-test-slave-9q7f krb5kdc[13751](info): commencing operation
krb5kdc: starting...
W20260923 17:23:52.504316 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:52 dist-test-slave-9q7f krb5kdc[13751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:52.629585 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38367
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:52.740988 13764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:52.741345 13764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:52.741429 13764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:52.745501 13764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:52.745630 13764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:52.745671 13764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:52.745705 13764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:52.745734 13764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:52.751300 13764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:38367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:52.752480 13764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:52.753575 13764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:52.762174 13769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:52.762235 13764 server_base.cc:1034] running on GCE node
W20260923 17:23:52.762189 13770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:52.762197 13772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:52.763002 13764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:52.764179 13764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:52.765378 13764 hybrid_clock.cc:648] HybridClock initialized: now 1790184232765349 us; error 44 us; skew 500 ppm
Sep 23 17:23:52 dist-test-slave-9q7f krb5kdc[13751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:52.775859 13764 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:52.776839 13764 webserver.cc:460] Webserver started at http://127.23.214.190:37441/ using document root <none> and password file <none>
I20260923 17:23:52.777450 13764 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:52.777549 13764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:52.777786 13764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:52.779918 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "8f523573b93744fdacddaabda08919ab"
format_stamp: "Formatted at 2026-09-23 17:23:52 on dist-test-slave-9q7f"
server_key: "956979ff747ea12b924014d9f2b23e76"
server_key_iv: "bdf8dff72a084b2643427152508e2743"
server_key_version: "encryptionkey@0"
I20260923 17:23:52.780488 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "8f523573b93744fdacddaabda08919ab"
format_stamp: "Formatted at 2026-09-23 17:23:52 on dist-test-slave-9q7f"
server_key: "956979ff747ea12b924014d9f2b23e76"
server_key_iv: "bdf8dff72a084b2643427152508e2743"
server_key_version: "encryptionkey@0"
I20260923 17:23:52.784240 13764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:23:52.786774 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:52.788127 13764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:52.788275 13764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "8f523573b93744fdacddaabda08919ab"
format_stamp: "Formatted at 2026-09-23 17:23:52 on dist-test-slave-9q7f"
server_key: "956979ff747ea12b924014d9f2b23e76"
server_key_iv: "bdf8dff72a084b2643427152508e2743"
server_key_version: "encryptionkey@0"
I20260923 17:23:52.788399 13764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:52.801465 13764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:52.804306 13764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:52.804523 13764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:52.811754 13764 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:38367
I20260923 17:23:52.811779 13831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:38367 every 8 connection(s)
I20260923 17:23:52.812705 13764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:52.815804 13832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:52.816092 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13764
I20260923 17:23:52.816231 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:52.816527 24410 external_mini_cluster.cc:1428] Setting key bf4353d55e548b01b86a3ef3d898145c
I20260923 17:23:52.823056 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab: Bootstrap starting.
I20260923 17:23:52.825933 13832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:52.826607 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:52.817965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52435 (local address 127.23.214.190:38367)
0923 17:23:52.818391 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:52.818411 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:52.818454 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:23:52.819223 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:52.819246 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:52.819686 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:52.820065 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:52.820465 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:52.822001 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:52.822889 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:52.823552 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:52.823706 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:52.824590 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:52.824596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:52.824613 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:52.825105 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:52.825170 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:52.825201 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:52.825328 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:52.825485 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 17:23:52.825620 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
I20260923 17:23:52.826799 13832 log.cc:824] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:52.828980 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab: No bootstrap required, opened a new log
I20260923 17:23:52.829632 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:52.832902 13832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f523573b93744fdacddaabda08919ab" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38367 } }
I20260923 17:23:52.833127 13832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:52.833190 13832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f523573b93744fdacddaabda08919ab, State: Initialized, Role: FOLLOWER
I20260923 17:23:52.833604 13832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f523573b93744fdacddaabda08919ab" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38367 } }
I20260923 17:23:52.833724 13832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:52.833811 13832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:52.833957 13832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:52.834874 13832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f523573b93744fdacddaabda08919ab" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38367 } }
I20260923 17:23:52.835276 13832 leader_election.cc:302] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f523573b93744fdacddaabda08919ab; no voters:
I20260923 17:23:52.835691 13832 leader_election.cc:288] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:52.835739 13842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:52.835969 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:52.831678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36847 (local address 127.23.214.190:38367)
0923 17:23:52.831817 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:52.831824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:52.832014 (+ 190us) server_negotiation.cc:407] Connection header received
0923 17:23:52.832151 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:52.832154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:52.832199 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:52.832286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:52.832554 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:52.833128 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:52.834247 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:52.834521 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:52.834640 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:52.835473 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:52.835476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:52.835479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:52.835492 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:52.835525 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:52.835537 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:52.835635 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:52.835776 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 17:23:52.835812 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:23:52.836035 13842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [term 1 LEADER]: Becoming Leader. State: Replica: 8f523573b93744fdacddaabda08919ab, State: Running, Role: LEADER
I20260923 17:23:52.836625 13842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f523573b93744fdacddaabda08919ab" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38367 } }
I20260923 17:23:52.837164 13832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:52.837850 13842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f523573b93744fdacddaabda08919ab. Latest consensus state: current_term: 1 leader_uuid: "8f523573b93744fdacddaabda08919ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f523573b93744fdacddaabda08919ab" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38367 } } }
I20260923 17:23:52.838042 13842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:52.838258 13842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f523573b93744fdacddaabda08919ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f523573b93744fdacddaabda08919ab" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 38367 } } }
I20260923 17:23:52.838414 13842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:52.838410 13850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:52.842644 13850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:52.848618 13850 catalog_manager.cc:1347] Generated new cluster ID: c063704bdaa24033a757da2b09c432d4
I20260923 17:23:52.848701 13850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:52.862262 13850 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:52.863590 13850 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:52.872041 13850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f523573b93744fdacddaabda08919ab: Generated new TSK 0
I20260923 17:23:52.872895 13850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:52.878873 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13764
W20260923 17:23:52.884303 13749 connection.cc:535] client connection to 127.23.214.190:38367 recv error: Network error: failed to read from TLS socket (remote: 127.23.214.190:38367): Connection reset by peer (error 104)
2026-09-23T17:23:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:52 dist-test-slave-9q7f krb5kdc[13863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:52 dist-test-slave-9q7f krb5kdc[13863](info): set up 2 sockets
Sep 23 17:23:52 dist-test-slave-9q7f krb5kdc[13863](info): commencing operation
krb5kdc: starting...
W20260923 17:23:53.928236 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:54 dist-test-slave-9q7f krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:54.052529 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35075
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:35075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:54.161926 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:54.162271 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:54.162359 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:54.166443 13876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:54.166530 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:54.166569 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:54.166625 13876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:54.166662 13876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:54.172134 13876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:35075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:54.173252 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:54.174321 13876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:54.183045 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:54.183038 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:54.183106 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:54.183228 13876 server_base.cc:1034] running on GCE node
I20260923 17:23:54.183859 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:54.184962 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:54.186118 13876 hybrid_clock.cc:648] HybridClock initialized: now 1790184234186104 us; error 28 us; skew 500 ppm
Sep 23 17:23:54 dist-test-slave-9q7f krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:54.188984 13876 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:54.189855 13876 webserver.cc:460] Webserver started at http://127.23.214.190:32917/ using document root <none> and password file <none>
I20260923 17:23:54.190433 13876 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:54.190491 13876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:54.190649 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:54.192633 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "6d709ca761b34dc2bf26390ab05c5111"
format_stamp: "Formatted at 2026-09-23 17:23:54 on dist-test-slave-9q7f"
server_key: "27ea48787c7aedf10c097bb58ea5a1ff"
server_key_iv: "47ba9728a3140f9dcccd08db07aa93a6"
server_key_version: "encryptionkey@0"
I20260923 17:23:54.193186 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "6d709ca761b34dc2bf26390ab05c5111"
format_stamp: "Formatted at 2026-09-23 17:23:54 on dist-test-slave-9q7f"
server_key: "27ea48787c7aedf10c097bb58ea5a1ff"
server_key_iv: "47ba9728a3140f9dcccd08db07aa93a6"
server_key_version: "encryptionkey@0"
I20260923 17:23:54.197124 13876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 17:23:54.199580 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:54.200913 13876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 17:23:54.201053 13876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "6d709ca761b34dc2bf26390ab05c5111"
format_stamp: "Formatted at 2026-09-23 17:23:54 on dist-test-slave-9q7f"
server_key: "27ea48787c7aedf10c097bb58ea5a1ff"
server_key_iv: "47ba9728a3140f9dcccd08db07aa93a6"
server_key_version: "encryptionkey@0"
I20260923 17:23:54.201169 13876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:54.205689 13876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:54.208940 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:54.209136 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:54.216069 13876 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:35075
I20260923 17:23:54.216095 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:35075 every 8 connection(s)
I20260923 17:23:54.216989 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:54.218668 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13876
I20260923 17:23:54.218775 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:54.219089 24410 external_mini_cluster.cc:1428] Setting key 0dc062525650c7db2623519fa48f8bd5
I20260923 17:23:54.219901 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:54.224840 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:54.220691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44171 (local address 127.23.214.190:35075)
0923 17:23:54.221167 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:54.221182 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:54.221226 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:23:54.222031 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:54.222050 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:54.222348 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:54.222715 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:54.223567 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:54.223571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:54.223583 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:54.223594 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:54.223652 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:54.223672 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:54.223850 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:54.223926 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 17:23:54.224095 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":165,"threads_started":1}
I20260923 17:23:54.226615 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111: Bootstrap starting.
I20260923 17:23:54.227905 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:54.229126 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:54.229969 13944 log.cc:824] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:54.231926 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:54.229982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39335 (local address 127.23.214.190:35075)
0923 17:23:54.230119 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:54.230123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:54.230558 (+ 435us) server_negotiation.cc:407] Connection header received
0923 17:23:54.230636 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:54.230639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:54.230680 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:54.230769 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:54.231510 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:54.231513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:54.231515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:54.231525 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:54.231548 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:54.231558 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:54.231624 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:54.231742 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 17:23:54.231774 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:23:54.232329 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111: No bootstrap required, opened a new log
I20260923 17:23:54.235869 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d709ca761b34dc2bf26390ab05c5111" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35075 } }
I20260923 17:23:54.236124 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:54.236209 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d709ca761b34dc2bf26390ab05c5111, State: Initialized, Role: FOLLOWER
I20260923 17:23:54.236635 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d709ca761b34dc2bf26390ab05c5111" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35075 } }
I20260923 17:23:54.236788 13944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:54.236881 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:54.236980 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:54.237951 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d709ca761b34dc2bf26390ab05c5111" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35075 } }
I20260923 17:23:54.238305 13944 leader_election.cc:302] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d709ca761b34dc2bf26390ab05c5111; no voters:
I20260923 17:23:54.238638 13944 leader_election.cc:288] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:54.238775 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:54.239029 13954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [term 1 LEADER]: Becoming Leader. State: Replica: 6d709ca761b34dc2bf26390ab05c5111, State: Running, Role: LEADER
I20260923 17:23:54.239387 13954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d709ca761b34dc2bf26390ab05c5111" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35075 } }
I20260923 17:23:54.239923 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:54.241084 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d709ca761b34dc2bf26390ab05c5111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d709ca761b34dc2bf26390ab05c5111" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35075 } } }
I20260923 17:23:54.241218 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:54.241580 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d709ca761b34dc2bf26390ab05c5111. Latest consensus state: current_term: 1 leader_uuid: "6d709ca761b34dc2bf26390ab05c5111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d709ca761b34dc2bf26390ab05c5111" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35075 } } }
I20260923 17:23:54.241688 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111 [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:54.241737 13963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:54.244463 13963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:54.250566 13963 catalog_manager.cc:1347] Generated new cluster ID: e00e0663bb824e30ae06daba4144db21
I20260923 17:23:54.250653 13963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:54.261754 13963 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:54.262125 13963 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:54.270486 13963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d709ca761b34dc2bf26390ab05c5111: Generated new TSK 0
I20260923 17:23:54.271288 13963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:54.284569 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13876
2026-09-23T17:23:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:54 dist-test-slave-9q7f krb5kdc[13975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:54 dist-test-slave-9q7f krb5kdc[13975](info): set up 2 sockets
Sep 23 17:23:54 dist-test-slave-9q7f krb5kdc[13975](info): commencing operation
krb5kdc: starting...
W20260923 17:23:55.321099 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:55 dist-test-slave-9q7f krb5kdc[13975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:55.447063 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:39383
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:42167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:55.556197 13988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:55.556528 13988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:55.556615 13988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:55.560730 13988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:55.560842 13988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:55.560881 13988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:55.560935 13988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:55.560974 13988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:55.566635 13988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:42167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:39383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:55.567795 13988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:55.569167 13988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:55.578271 13993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:55.578366 13994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:55.578270 13996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:55.578542 13988 server_base.cc:1034] running on GCE node
I20260923 17:23:55.579146 13988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:55.580211 13988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:55.581401 13988 hybrid_clock.cc:648] HybridClock initialized: now 1790184235581390 us; error 23 us; skew 500 ppm
Sep 23 17:23:55 dist-test-slave-9q7f krb5kdc[13975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184235, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:55.584149 13988 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:55.584966 13988 webserver.cc:460] Webserver started at http://127.23.214.190:40963/ using document root <none> and password file <none>
I20260923 17:23:55.585505 13988 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:55.585595 13988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:55.585809 13988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:55.587764 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "1b62a8a017e84472b3707184803680ca"
format_stamp: "Formatted at 2026-09-23 17:23:55 on dist-test-slave-9q7f"
server_key: "8545de9e44453100d8f97b1c43723719"
server_key_iv: "12c523a241e98999742d6173c9863645"
server_key_version: "encryptionkey@0"
I20260923 17:23:55.588312 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "1b62a8a017e84472b3707184803680ca"
format_stamp: "Formatted at 2026-09-23 17:23:55 on dist-test-slave-9q7f"
server_key: "8545de9e44453100d8f97b1c43723719"
server_key_iv: "12c523a241e98999742d6173c9863645"
server_key_version: "encryptionkey@0"
I20260923 17:23:55.591930 13988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:23:55.594197 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:55.595366 13988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 17:23:55.595499 13988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "1b62a8a017e84472b3707184803680ca"
format_stamp: "Formatted at 2026-09-23 17:23:55 on dist-test-slave-9q7f"
server_key: "8545de9e44453100d8f97b1c43723719"
server_key_iv: "12c523a241e98999742d6173c9863645"
server_key_version: "encryptionkey@0"
I20260923 17:23:55.595607 13988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:55.614315 13988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:55.617126 13988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:55.617344 13988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:55.624591 13988 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:39383
I20260923 17:23:55.624620 14055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:39383 every 8 connection(s)
I20260923 17:23:55.625537 13988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:55.628468 14056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:55.633646 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 13988
I20260923 17:23:55.633791 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:55.634110 24410 external_mini_cluster.cc:1428] Setting key af6ff4b46e6f1b2af2d3513669581d33
I20260923 17:23:55.634163 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca: Bootstrap starting.
I20260923 17:23:55.636616 14056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:55.637355 14056 log.cc:824] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:55.639262 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca: No bootstrap required, opened a new log
Sep 23 17:23:55 dist-test-slave-9q7f krb5kdc[13975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:55.643230 14056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b62a8a017e84472b3707184803680ca" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39383 } }
I20260923 17:23:55.643453 14056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:55.643512 14056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b62a8a017e84472b3707184803680ca, State: Initialized, Role: FOLLOWER
I20260923 17:23:55.644057 14056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b62a8a017e84472b3707184803680ca" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39383 } }
I20260923 17:23:55.644238 14056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:55.644318 14056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:55.644443 14056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:55.645457 14056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b62a8a017e84472b3707184803680ca" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39383 } }
I20260923 17:23:55.645824 14056 leader_election.cc:302] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b62a8a017e84472b3707184803680ca; no voters:
I20260923 17:23:55.646214 14056 leader_election.cc:288] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:55.646411 14061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:55.646616 14061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [term 1 LEADER]: Becoming Leader. State: Replica: 1b62a8a017e84472b3707184803680ca, State: Running, Role: LEADER
I20260923 17:23:55.647004 14061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b62a8a017e84472b3707184803680ca" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39383 } }
I20260923 17:23:55.647670 14056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:55.648717 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b62a8a017e84472b3707184803680ca. Latest consensus state: current_term: 1 leader_uuid: "1b62a8a017e84472b3707184803680ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b62a8a017e84472b3707184803680ca" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39383 } } }
I20260923 17:23:55.648845 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:55.649124 14062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b62a8a017e84472b3707184803680ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b62a8a017e84472b3707184803680ca" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 39383 } } }
I20260923 17:23:55.649183 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:55.635455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53229 (local address 127.23.214.190:39383)
0923 17:23:55.635892 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:55.635907 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:55.635940 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:23:55.636543 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:55.636559 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:55.636951 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:55.637303 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:55.638185 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:55.639141 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:55.640067 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:55.640534 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:55.640708 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:55.643647 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:55.643652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:55.643669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:55.643706 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:55.645605 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:55.646125 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:55.646131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:55.646137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:55.646311 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:55.646657 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:55.646662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:55.646664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:55.647173 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:55.647377 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:55.647624 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 17:23:55.648063 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260923 17:23:55.649205 14062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:55.649240 14070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:55.652194 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:55.652835 14070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:55.659096 14070 catalog_manager.cc:1347] Generated new cluster ID: 9b60eeef7c644a389ced60b2fa4bd18b
I20260923 17:23:55.659161 14070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:55.663240 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:55.654724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46633 (local address 127.23.214.190:39383)
0923 17:23:55.654862 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:55.654868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:55.655075 (+ 207us) server_negotiation.cc:407] Connection header received
0923 17:23:55.655211 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:55.655214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:55.655257 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:55.655354 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:55.656016 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:55.656532 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:55.657429 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:55.657926 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:55.658033 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:55.659761 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:55.659765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:55.659767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:55.659794 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:55.661464 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:55.662047 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:55.662051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:55.662053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:55.662131 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:55.662421 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:55.662424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:55.662425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:55.662607 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:55.662728 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:55.662984 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 17:23:55.663088 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:23:55.668792 14070 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:55.670046 14070 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:55.684140 14070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b62a8a017e84472b3707184803680ca: Generated new TSK 0
I20260923 17:23:55.684894 14070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:55.697688 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 13988
2026-09-23T17:23:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:55 dist-test-slave-9q7f krb5kdc[14087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:55 dist-test-slave-9q7f krb5kdc[14087](info): set up 2 sockets
Sep 23 17:23:55 dist-test-slave-9q7f krb5kdc[14087](info): commencing operation
krb5kdc: starting...
W20260923 17:23:56.728382 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:56 dist-test-slave-9q7f krb5kdc[14087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:56.852819 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35377
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:35377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:56.967723 14100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:56.968046 14100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:56.968119 14100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:56.972201 14100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:56.972308 14100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:56.972348 14100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:56.972406 14100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:56.972440 14100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:56.977742 14100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:35377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:35377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:56.978844 14100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:56.979861 14100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:56.988566 14108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:56.988623 14100 server_base.cc:1034] running on GCE node
W20260923 17:23:56.988574 14105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:56.988871 14106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:56.989408 14100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:56.990569 14100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:56.991730 14100 hybrid_clock.cc:648] HybridClock initialized: now 1790184236991723 us; error 26 us; skew 500 ppm
Sep 23 17:23:56 dist-test-slave-9q7f krb5kdc[14087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184236, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:56.994402 14100 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:56.995223 14100 webserver.cc:460] Webserver started at http://127.23.214.190:46067/ using document root <none> and password file <none>
I20260923 17:23:56.995783 14100 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:56.995880 14100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:56.996127 14100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:56.998220 14100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "4e7b00ff36494685ae033ea143c5c4db"
format_stamp: "Formatted at 2026-09-23 17:23:56 on dist-test-slave-9q7f"
server_key: "be634f865815d2f681b0abd2d3e54561"
server_key_iv: "e0ee464b7a8b45efef33f9cabcdd60b5"
server_key_version: "encryptionkey@0"
I20260923 17:23:56.998809 14100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "4e7b00ff36494685ae033ea143c5c4db"
format_stamp: "Formatted at 2026-09-23 17:23:56 on dist-test-slave-9q7f"
server_key: "be634f865815d2f681b0abd2d3e54561"
server_key_iv: "e0ee464b7a8b45efef33f9cabcdd60b5"
server_key_version: "encryptionkey@0"
I20260923 17:23:57.002482 14100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:23:57.004838 14115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:57.006134 14100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:57.006278 14100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "4e7b00ff36494685ae033ea143c5c4db"
format_stamp: "Formatted at 2026-09-23 17:23:56 on dist-test-slave-9q7f"
server_key: "be634f865815d2f681b0abd2d3e54561"
server_key_iv: "e0ee464b7a8b45efef33f9cabcdd60b5"
server_key_version: "encryptionkey@0"
I20260923 17:23:57.006402 14100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:57.018977 14100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:57.021689 14100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:57.021946 14100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:57.028785 14100 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:35377
I20260923 17:23:57.028811 14167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:35377 every 8 connection(s)
I20260923 17:23:57.029785 14100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:57.033509 14168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:57.039309 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14100
I20260923 17:23:57.039472 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:57.039793 24410 external_mini_cluster.cc:1428] Setting key 944965ac723ff8dcab9a81f8f9cf6f4b
I20260923 17:23:57.041828 14168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db: Bootstrap starting.
I20260923 17:23:57.044176 14168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:57.045074 14168 log.cc:824] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:57.047401 14168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db: No bootstrap required, opened a new log
I20260923 17:23:57.048923 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:57.041143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37453 (local address 127.23.214.190:35377)
0923 17:23:57.041552 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:57.041570 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:57.041615 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:23:57.042352 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:57.042370 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:57.042701 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:57.042977 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:57.043288 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:57.044304 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:57.045511 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:57.046183 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:57.046303 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:57.047183 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:57.047187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:57.047199 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:57.047642 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:57.047699 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:57.047722 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:57.047799 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:57.047961 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 17:23:57.048190 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":141,"threads_started":1}
I20260923 17:23:57.051359 14168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b00ff36494685ae033ea143c5c4db" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35377 } }
I20260923 17:23:57.051622 14168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:57.051719 14168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7b00ff36494685ae033ea143c5c4db, State: Initialized, Role: FOLLOWER
I20260923 17:23:57.051817 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:57.052224 14168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b00ff36494685ae033ea143c5c4db" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35377 } }
I20260923 17:23:57.052387 14168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:57.052455 14168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:57.052553 14168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:57.053645 14168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b00ff36494685ae033ea143c5c4db" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35377 } }
I20260923 17:23:57.054112 14168 leader_election.cc:302] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e7b00ff36494685ae033ea143c5c4db; no voters:
I20260923 17:23:57.054471 14168 leader_election.cc:288] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:57.054569 14178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:57.054785 14178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [term 1 LEADER]: Becoming Leader. State: Replica: 4e7b00ff36494685ae033ea143c5c4db, State: Running, Role: LEADER
I20260923 17:23:57.055186 14178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b00ff36494685ae033ea143c5c4db" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35377 } }
I20260923 17:23:57.055737 14168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:57.056787 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e7b00ff36494685ae033ea143c5c4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b00ff36494685ae033ea143c5c4db" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35377 } } }
I20260923 17:23:57.056979 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:57.057355 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e7b00ff36494685ae033ea143c5c4db. Latest consensus state: current_term: 1 leader_uuid: "4e7b00ff36494685ae033ea143c5c4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b00ff36494685ae033ea143c5c4db" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 35377 } } }
I20260923 17:23:57.057493 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:57.058887 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:57.053854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53515 (local address 127.23.214.190:35377)
0923 17:23:57.053999 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:57.054003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:57.054167 (+ 164us) server_negotiation.cc:407] Connection header received
0923 17:23:57.054253 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:57.054256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:57.054311 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:57.054402 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:57.054785 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:57.055551 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:57.056782 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:57.057171 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:57.057250 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:57.058411 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:57.058415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:57.058418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:57.058432 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:57.058460 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:57.058473 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:57.058520 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:57.058684 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 17:23:57.058736 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
W20260923 17:23:57.060981 14194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:23:57.061074 14194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:23:57.061146 14187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:57.061868 14187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:57.067543 14187 catalog_manager.cc:1347] Generated new cluster ID: 8ba19478b2554f4fb037bc4fafcd872e
I20260923 17:23:57.067616 14187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:57.077015 14187 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:57.078301 14187 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:57.089315 14187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e7b00ff36494685ae033ea143c5c4db: Generated new TSK 0
I20260923 17:23:57.090166 14187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:57.097600 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14100
2026-09-23T17:23:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:57 dist-test-slave-9q7f krb5kdc[14199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:57 dist-test-slave-9q7f krb5kdc[14199](info): set up 2 sockets
Sep 23 17:23:57 dist-test-slave-9q7f krb5kdc[14199](info): commencing operation
krb5kdc: starting...
W20260923 17:23:58.136251 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:58 dist-test-slave-9q7f krb5kdc[14199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:58.257068 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44823
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:44823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:58.365518 14212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:58.365809 14212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:58.365854 14212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:58.369947 14212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:58.370016 14212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:58.370036 14212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:58.370054 14212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:58.370069 14212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:58.375530 14212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:44823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:58.376603 14212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:58.377583 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:23:58.386278 14212 server_base.cc:1034] running on GCE node
W20260923 17:23:58.386260 14218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:58.386212 14220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:58.386217 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:58.387084 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:58.388203 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:58.389382 14212 hybrid_clock.cc:648] HybridClock initialized: now 1790184238389364 us; error 30 us; skew 500 ppm
Sep 23 17:23:58 dist-test-slave-9q7f krb5kdc[14199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184238, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:58.397802 14212 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:58.398855 14212 webserver.cc:460] Webserver started at http://127.23.214.190:34749/ using document root <none> and password file <none>
I20260923 17:23:58.399402 14212 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:58.399461 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:58.399629 14212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:58.401659 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "460f4820b1704e048247d3a8c3eda0da"
format_stamp: "Formatted at 2026-09-23 17:23:58 on dist-test-slave-9q7f"
server_key: "ee1098bfc248695dff1126e867bae871"
server_key_iv: "617f25fae7e326fc3214e53bf8e6175b"
server_key_version: "encryptionkey@0"
I20260923 17:23:58.402246 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "460f4820b1704e048247d3a8c3eda0da"
format_stamp: "Formatted at 2026-09-23 17:23:58 on dist-test-slave-9q7f"
server_key: "ee1098bfc248695dff1126e867bae871"
server_key_iv: "617f25fae7e326fc3214e53bf8e6175b"
server_key_version: "encryptionkey@0"
I20260923 17:23:58.405831 14212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:23:58.408186 14227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:58.409426 14212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:23:58.409556 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "460f4820b1704e048247d3a8c3eda0da"
format_stamp: "Formatted at 2026-09-23 17:23:58 on dist-test-slave-9q7f"
server_key: "ee1098bfc248695dff1126e867bae871"
server_key_iv: "617f25fae7e326fc3214e53bf8e6175b"
server_key_version: "encryptionkey@0"
I20260923 17:23:58.409659 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:58.414350 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:58.417600 14212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:58.417820 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:58.425189 14212 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:44823
I20260923 17:23:58.425216 14279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:44823 every 8 connection(s)
I20260923 17:23:58.426149 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:58.428817 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:58.432767 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14212
I20260923 17:23:58.432881 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:58.433161 24410 external_mini_cluster.cc:1428] Setting key c43ab295e8624377d53b0cc24d90c25b
I20260923 17:23:58.434952 14280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da: Bootstrap starting.
I20260923 17:23:58.437672 14280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da: Neither blocks nor log segments found. Creating new log.
I20260923 17:23:58.438562 14280 log.cc:824] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da: Log is configured to *not* fsync() on all Append() calls
I20260923 17:23:58.438813 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:58.434634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37735 (local address 127.23.214.190:44823)
0923 17:23:58.435066 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:58.435085 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:58.435127 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:23:58.435931 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:58.435955 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:58.436346 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:58.436699 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:58.437416 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:58.437421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:58.437435 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:58.437447 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:58.437511 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:58.437541 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:58.437658 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:58.437732 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 17:23:58.437913 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260923 17:23:58.440662 14280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da: No bootstrap required, opened a new log
I20260923 17:23:58.441835 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:23:58.445500 14280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460f4820b1704e048247d3a8c3eda0da" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44823 } }
I20260923 17:23:58.445822 14280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:23:58.445916 14280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460f4820b1704e048247d3a8c3eda0da, State: Initialized, Role: FOLLOWER
I20260923 17:23:58.446040 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:58.444158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41687 (local address 127.23.214.190:44823)
0923 17:23:58.444306 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:58.444313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:58.444522 (+ 209us) server_negotiation.cc:407] Connection header received
0923 17:23:58.444647 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:58.444649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:58.444681 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:58.444779 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:58.445512 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:58.445515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:58.445517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:23:58.445528 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:58.445553 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:23:58.445563 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:58.445636 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:58.445752 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 17:23:58.445818 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:23:58.446478 14280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460f4820b1704e048247d3a8c3eda0da" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44823 } }
I20260923 17:23:58.446640 14280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:23:58.446707 14280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:23:58.446812 14280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:23:58.447759 14280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460f4820b1704e048247d3a8c3eda0da" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44823 } }
I20260923 17:23:58.448117 14280 leader_election.cc:302] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460f4820b1704e048247d3a8c3eda0da; no voters:
I20260923 17:23:58.448431 14280 leader_election.cc:288] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:23:58.448568 14290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:23:58.448812 14290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [term 1 LEADER]: Becoming Leader. State: Replica: 460f4820b1704e048247d3a8c3eda0da, State: Running, Role: LEADER
I20260923 17:23:58.449182 14290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460f4820b1704e048247d3a8c3eda0da" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44823 } }
I20260923 17:23:58.449735 14280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:23:58.450999 14291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "460f4820b1704e048247d3a8c3eda0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460f4820b1704e048247d3a8c3eda0da" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44823 } } }
I20260923 17:23:58.451136 14291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:58.451472 14292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460f4820b1704e048247d3a8c3eda0da. Latest consensus state: current_term: 1 leader_uuid: "460f4820b1704e048247d3a8c3eda0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460f4820b1704e048247d3a8c3eda0da" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 44823 } } }
I20260923 17:23:58.451581 14292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da [sys.catalog]: This master's current role is: LEADER
I20260923 17:23:58.452157 14300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:23:58.455122 14300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:23:58.461338 14300 catalog_manager.cc:1347] Generated new cluster ID: 46eea13a8804478b9ffb537aadcc7ee0
I20260923 17:23:58.461436 14300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:23:58.469584 14300 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:23:58.469971 14300 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:23:58.474895 14300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 460f4820b1704e048247d3a8c3eda0da: Generated new TSK 0
I20260923 17:23:58.475699 14300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:23:58.479946 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14212
2026-09-23T17:23:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1382 ms)
[----------] 18 tests from AuthTokenIssuingTest (25530 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:23:58 dist-test-slave-9q7f krb5kdc[14311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:23:58 dist-test-slave-9q7f krb5kdc[14311](info): set up 2 sockets
Sep 23 17:23:58 dist-test-slave-9q7f krb5kdc[14311](info): commencing operation
krb5kdc: starting...
W20260923 17:23:59.520298 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:23:59 dist-test-slave-9q7f krb5kdc[14311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:23:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:23:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:23:59.644755 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:37869
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:37869,127.23.214.189:33017,127.23.214.188:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:23:59.755896 14324 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:23:59.756246 14324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:59.756319 14324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:59.756395 14324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:23:59.760412 14324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:59.760509 14324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:59.760548 14324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:59.760581 14324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:59.760607 14324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:59.765846 14324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:37869,127.23.214.189:33017,127.23.214.188:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:59.766959 14324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:59.767995 14324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:59.776553 14329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:59.776574 14330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:59.776618 14332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:59.776749 14324 server_base.cc:1034] running on GCE node
I20260923 17:23:59.777352 14324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:59.778450 14324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:59.779642 14324 hybrid_clock.cc:648] HybridClock initialized: now 1790184239779626 us; error 25 us; skew 500 ppm
Sep 23 17:23:59 dist-test-slave-9q7f krb5kdc[14311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:59.782447 14324 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:23:59.783275 14324 webserver.cc:460] Webserver started at http://127.23.214.190:35783/ using document root <none> and password file <none>
I20260923 17:23:59.783833 14324 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:59.783924 14324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:59.784147 14324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:59.786242 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "19041f41db804d16ae0ebf40d23356d4"
format_stamp: "Formatted at 2026-09-23 17:23:59 on dist-test-slave-9q7f"
server_key: "2266bfd104f692f8602a2cf22b5f535c"
server_key_iv: "86a2421b7d25c10b8af9aeef87b7f61d"
server_key_version: "encryptionkey@0"
I20260923 17:23:59.786826 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "19041f41db804d16ae0ebf40d23356d4"
format_stamp: "Formatted at 2026-09-23 17:23:59 on dist-test-slave-9q7f"
server_key: "2266bfd104f692f8602a2cf22b5f535c"
server_key_iv: "86a2421b7d25c10b8af9aeef87b7f61d"
server_key_version: "encryptionkey@0"
I20260923 17:23:59.790527 14324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:23:59.792819 14339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:23:59.794082 14324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:23:59.794212 14324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "19041f41db804d16ae0ebf40d23356d4"
format_stamp: "Formatted at 2026-09-23 17:23:59 on dist-test-slave-9q7f"
server_key: "2266bfd104f692f8602a2cf22b5f535c"
server_key_iv: "86a2421b7d25c10b8af9aeef87b7f61d"
server_key_version: "encryptionkey@0"
I20260923 17:23:59.794320 14324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:23:59.803035 14324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:23:59.805755 14324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:23:59.805987 14324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:23:59.812803 14324 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:37869
I20260923 17:23:59.813074 14391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:37869 every 8 connection(s)
I20260923 17:23:59.813902 14324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:23:59.816710 14392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:23:59.820470 14392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:23:59.821117 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14324
I20260923 17:23:59.821280 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:23:59.821590 24410 external_mini_cluster.cc:1428] Setting key 084c95fb2edcb8d24a0006d801757976
WARNING: no policy specified for kudu/127.23.214.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.189@KRBTEST.COM" created.
Sep 23 17:23:59 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:23:59.839901 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.823143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.23.214.190:37869)
0923 17:23:59.823710 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0923 17:23:59.823742 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0923 17:23:59.823920 (+ 178us) server_negotiation.cc:407] Connection header received
0923 17:23:59.825442 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:23:59.825453 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:23:59.825642 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:23:59.825834 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:23:59.827135 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:59.828023 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:59.829550 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:23:59.829997 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:23:59.830118 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:59.834446 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:23:59.834453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:23:59.834463 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:23:59.834485 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:23:59.836536 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:59.837063 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:59.837069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:59.837071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:59.837192 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:23:59.837512 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:23:59.837517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:23:59.837519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:23:59.838003 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:23:59.838153 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:23:59.838531 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 17:23:59.838783 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"spinlock_wait_cycles":31616,"thread_start_us":162,"threads_started":1}
I20260923 17:23:59.839901 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.823143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:37869 (local address 127.0.0.1:54498)
0923 17:23:59.823725 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:59.823786 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:23:59.825119 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:23:59.826031 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:23:59.826040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:23:59.826550 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:23:59.826961 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:59.826986 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:59.828159 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:59.828163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:59.829418 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:23:59.829429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:23:59.830153 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:23:59.830157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:23:59.830287 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:23:59.830910 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:23:59.830926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:23:59.834265 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:23:59.836663 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:59.836701 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:59.836716 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:59.836943 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:59.837296 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:23:59.837299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:23:59.837301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:23:59.837416 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:23:59.838120 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:23:59.838127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:23:59.838277 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 17:23:59.838455 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 17:23:59.838851 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":65,"thread_start_us":140,"threads_started":1}
I20260923 17:23:59.842530 14392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:23:59.843326 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.843038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59316)
0923 17:23:59.843177 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:59.843258 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:23:59.843528 14343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
I20260923 17:23:59.844420 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.844206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59324)
0923 17:23:59.844317 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:59.844366 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:23:59.844559 14392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:33017: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab.
Entry for principal kudu/127.23.214.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab.
I20260923 17:23:59.849403 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:33017
--webserver_interface=127.23.214.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab
--principal=kudu/127.23.214.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:37869,127.23.214.189:33017,127.23.214.188:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:23:59.893821 14392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } attempt: 1
I20260923 17:23:59.894733 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.894434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59338)
0923 17:23:59.894585 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:59.894655 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 17:23:59.895535 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.895304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59348)
0923 17:23:59.895418 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:59.895469 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:23:59.895720 14392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:33017: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
W20260923 17:23:59.961122 14399 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:23:59.961558 14399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:23:59.961639 14399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:23:59.961735 14399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:23:59.963897 14392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } attempt: 2
I20260923 17:23:59.964751 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.964460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59350)
0923 17:23:59.964613 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:59.964676 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:23:59.965484 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:23:59.965246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59366)
0923 17:23:59.965379 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:23:59.965429 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 17:23:59.965618 14392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:33017: Network error: Client connection negotiation failed: client connection to 127.23.214.189:33017: connect: Connection refused (error 111)
W20260923 17:23:59.966869 14399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:23:59.966952 14399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:23:59.966997 14399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:23:59.967036 14399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:23:59.967072 14399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:23:59.972877 14399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:37869,127.23.214.189:33017,127.23.214.188:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab
--principal=kudu/127.23.214.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.214.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:23:59.974040 14399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:23:59.975090 14399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:23:59.983820 14405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:23:59.983892 14404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:59.983937 14399 server_base.cc:1034] running on GCE node
W20260923 17:23:59.983898 14407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:23:59.984651 14399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:23:59.985740 14399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:23:59.986934 14399 hybrid_clock.cc:648] HybridClock initialized: now 1790184239986919 us; error 26 us; skew 500 ppm
Sep 23 17:23:59 dist-test-slave-9q7f krb5kdc[14311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:23:59.990159 14399 init.cc:377] Logged in from keytab as kudu/127.23.214.189@KRBTEST.COM (short username kudu)
I20260923 17:23:59.991065 14399 webserver.cc:460] Webserver started at http://127.23.214.189:38903/ using document root <none> and password file <none>
I20260923 17:23:59.991638 14399 fs_manager.cc:359] Metadata directory not provided
I20260923 17:23:59.991732 14399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:23:59.991966 14399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:23:59.994102 14399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data/instance:
uuid: "7f70be81a86544ab9c582381523c16da"
format_stamp: "Formatted at 2026-09-23 17:23:59 on dist-test-slave-9q7f"
server_key: "dffd04796dbf6b750b7d614fa41db55e"
server_key_iv: "991202b41a6c86117a9f964af3d1d766"
server_key_version: "encryptionkey@0"
I20260923 17:23:59.994663 14399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/wal/instance:
uuid: "7f70be81a86544ab9c582381523c16da"
format_stamp: "Formatted at 2026-09-23 17:23:59 on dist-test-slave-9q7f"
server_key: "dffd04796dbf6b750b7d614fa41db55e"
server_key_iv: "991202b41a6c86117a9f964af3d1d766"
server_key_version: "encryptionkey@0"
I20260923 17:23:59.998384 14399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:24:00.000856 14414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:00.002136 14399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:00.002279 14399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/wal
uuid: "7f70be81a86544ab9c582381523c16da"
format_stamp: "Formatted at 2026-09-23 17:23:59 on dist-test-slave-9q7f"
server_key: "dffd04796dbf6b750b7d614fa41db55e"
server_key_iv: "991202b41a6c86117a9f964af3d1d766"
server_key_version: "encryptionkey@0"
I20260923 17:24:00.002393 14399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:00.016471 14399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:00.029101 14399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:00.029333 14399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:00.036226 14399 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.189:33017
I20260923 17:24:00.036520 14466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.189:33017 every 8 connection(s)
I20260923 17:24:00.037222 14399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
I20260923 17:24:00.040028 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:00.043738 14467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:00.046139 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14399
I20260923 17:24:00.046277 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-1/wal/instance
I20260923 17:24:00.046609 24410 external_mini_cluster.cc:1428] Setting key f5d72e534795415f21574b658e379f74
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.214.188@KRBTEST.COM; defaulting to no policy
I20260923 17:24:00.056778 14392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.188@KRBTEST.COM" created.
I20260923 17:24:00.060602 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.045610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.23.214.190:37869)
0923 17:24:00.045796 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.045801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.046828 (+ 1027us) server_negotiation.cc:407] Connection header received
0923 17:24:00.048303 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.048308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.048379 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.048473 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.049774 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.050352 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.052235 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.052495 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.052603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.055577 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.055580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.055582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.055608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.058088 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.058738 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.058744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.058749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.058839 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.059226 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.059232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.059237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.059501 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.059608 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.060236 (+ 628us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.060382 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:24:00.061537 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.046079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:37869 (local address 127.0.0.1:54514)
0923 17:24:00.046660 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.046717 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.048033 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.048626 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.048632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.048972 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.049611 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.049634 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.050497 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.050503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.052060 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.052077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.052682 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.052689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.052923 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.053585 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.053607 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.055368 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.058200 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.058206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.058222 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.058612 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.058941 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.058944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.058946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.059121 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.059616 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.059622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.059947 (+ 325us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.060153 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.060479 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":175,"threads_started":1}
I20260923 17:24:00.063090 14467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:24:00.068158 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.057294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59376)
0923 17:24:00.057534 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.057550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.057637 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.058364 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.058368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.058532 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.058674 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.058685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.059891 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.059894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.060667 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.060677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.061445 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.061452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.061595 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.062379 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.062398 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.064067 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.066394 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.066398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.066400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.066664 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.066972 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.066975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.066977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.067083 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.067612 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.067615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.067715 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.067838 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.067980 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260923 17:24:00.068231 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.057417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.23.214.189:33017)
0923 17:24:00.057809 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.057819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.057842 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:24:00.057969 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.057975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.058217 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.058379 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.058786 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.059757 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.060812 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.061312 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.061418 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.064212 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.064224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.064240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.064259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.066277 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.066775 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.066781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.066783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.066872 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.067187 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.067190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.067192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.067504 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.067647 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.067924 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.068058 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":197,"threads_started":1}
I20260923 17:24:00.070215 14392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:24:00.070973 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.070731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57924)
0923 17:24:00.070857 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.070906 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:24:00.071564 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.071330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57932)
0923 17:24:00.071467 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.071510 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260923 17:24:00.071702 14392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:43117: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
I20260923 17:24:00.074306 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.063651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59386)
0923 17:24:00.063764 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.063778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.063851 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.064277 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.064282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.064511 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.064650 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.064661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.065825 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.065832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.066451 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.066463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.067138 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.067144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.067257 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.068368 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.068381 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.070823 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.072907 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.072911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.072913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.073095 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.073417 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.073420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.073421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.073500 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.073831 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.073833 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.073961 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.074051 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.074165 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:24:00.074442 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.063785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.23.214.189:33017)
0923 17:24:00.064053 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.064061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.064071 (+ 10us) server_negotiation.cc:407] Connection header received
0923 17:24:00.064104 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.064106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.064163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.064244 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.065078 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.065702 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.066581 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.067017 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.067129 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.070966 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.070970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.070973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.070995 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.072782 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.073241 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.073244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.073246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.073324 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.073583 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.073585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.073586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.073752 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.073825 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.074104 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.074222 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
I20260923 17:24:00.075054 14467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:00.075686 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.075445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57938)
0923 17:24:00.075553 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.075629 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:24:00.075893 14418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab.
Entry for principal kudu/127.23.214.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab.
I20260923 17:24:00.076747 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.076513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57940)
0923 17:24:00.076629 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.076685 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:24:00.076898 14467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:43117: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
I20260923 17:24:00.077260 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:43117
--webserver_interface=127.23.214.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab
--principal=kudu/127.23.214.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:37869,127.23.214.189:33017,127.23.214.188:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:00.102876 14392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } attempt: 1
I20260923 17:24:00.103722 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.103428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57956)
0923 17:24:00.103575 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.103643 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:24:00.104421 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.104209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57968)
0923 17:24:00.104318 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.104367 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:24:00.104548 14392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:43117: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
I20260923 17:24:00.126155 14467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } attempt: 1
I20260923 17:24:00.127031 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.126745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57984)
0923 17:24:00.126893 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.126953 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:24:00.127734 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.127524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57988)
0923 17:24:00.127633 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.127683 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:24:00.127893 14467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:43117: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
I20260923 17:24:00.179687 14392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } attempt: 2
I20260923 17:24:00.180630 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.180312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57992)
0923 17:24:00.180470 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.180534 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 17:24:00.181645 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.181399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57996)
0923 17:24:00.181521 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.181573 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:24:00.181823 14392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:43117: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
I20260923 17:24:00.196043 14467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } attempt: 2
I20260923 17:24:00.197052 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.196619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:57998)
0923 17:24:00.196777 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.196850 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:24:00.197901 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.197610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:58002)
0923 17:24:00.197735 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.197790 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:24:00.198074 14467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:43117: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
W20260923 17:24:00.209792 14475 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:24:00.210402 14475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:00.210465 14475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:00.210558 14475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:00.219168 14475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:00.219240 14475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:00.219261 14475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:00.219278 14475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:00.219347 14475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:00.224993 14475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:37869,127.23.214.189:33017,127.23.214.188:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab
--principal=kudu/127.23.214.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.214.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:00.226208 14475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:00.227286 14475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:00.236003 14481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:00.236003 14483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:00.236016 14480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:00.236339 14475 server_base.cc:1034] running on GCE node
I20260923 17:24:00.236958 14475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:00.238205 14475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:00.239375 14475 hybrid_clock.cc:648] HybridClock initialized: now 1790184240239362 us; error 25 us; skew 500 ppm
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184240, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:00.242110 14475 init.cc:377] Logged in from keytab as kudu/127.23.214.188@KRBTEST.COM (short username kudu)
I20260923 17:24:00.243036 14475 webserver.cc:460] Webserver started at http://127.23.214.188:38477/ using document root <none> and password file <none>
I20260923 17:24:00.243597 14475 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:00.243660 14475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:00.243904 14475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:00.246090 14475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data/instance:
uuid: "3c0aa602adcd49e4a512a0cdb72ee459"
format_stamp: "Formatted at 2026-09-23 17:24:00 on dist-test-slave-9q7f"
server_key: "1a429f3d4bc8fcc25a36b133c43818e5"
server_key_iv: "09c1c9224ad311cd3b7e416ee495d11d"
server_key_version: "encryptionkey@0"
I20260923 17:24:00.246709 14475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/wal/instance:
uuid: "3c0aa602adcd49e4a512a0cdb72ee459"
format_stamp: "Formatted at 2026-09-23 17:24:00 on dist-test-slave-9q7f"
server_key: "1a429f3d4bc8fcc25a36b133c43818e5"
server_key_iv: "09c1c9224ad311cd3b7e416ee495d11d"
server_key_version: "encryptionkey@0"
I20260923 17:24:00.250736 14475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260923 17:24:00.253557 14490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:00.254906 14475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:00.255074 14475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/wal
uuid: "3c0aa602adcd49e4a512a0cdb72ee459"
format_stamp: "Formatted at 2026-09-23 17:24:00 on dist-test-slave-9q7f"
server_key: "1a429f3d4bc8fcc25a36b133c43818e5"
server_key_iv: "09c1c9224ad311cd3b7e416ee495d11d"
server_key_version: "encryptionkey@0"
I20260923 17:24:00.255236 14475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:00.273851 14475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:00.276990 14475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:00.277238 14475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:00.281019 14392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } attempt: 3
I20260923 17:24:00.281970 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.281558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:58012)
0923 17:24:00.281719 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.281779 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:24:00.282894 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.282576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:58014)
0923 17:24:00.282723 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.282794 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 17:24:00.283059 14392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:43117: Network error: Client connection negotiation failed: client connection to 127.23.214.188:43117: connect: Connection refused (error 111)
I20260923 17:24:00.285562 14475 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.188:43117
I20260923 17:24:00.285837 14542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.188:43117 every 8 connection(s)
I20260923 17:24:00.286720 14475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
I20260923 17:24:00.289268 14467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } attempt: 3
I20260923 17:24:00.289630 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:00.294036 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14475
I20260923 17:24:00.294170 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790183952943234-24410-0/minicluster-data/master-2/wal/instance
I20260923 17:24:00.294440 24410 external_mini_cluster.cc:1428] Setting key 3068b51761e2d6e8701c9b19ee1232cf
I20260923 17:24:00.294727 14543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184240, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:24:00.305092 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.289813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:58018)
0923 17:24:00.289981 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.289998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.290098 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.292759 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.292763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.293000 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.293126 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.293135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.294870 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.294874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.295638 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.295650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.296530 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.296537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.296631 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.297490 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.297510 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.299100 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.301963 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.301967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.301970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.302248 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.302680 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.302684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.302686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.302817 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.304500 (+ 1683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.304506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.304626 (+ 120us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.304743 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.304911 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 17:24:00.306536 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.290618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.23.214.188:43117)
0923 17:24:00.291119 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.291157 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.291196 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:24:00.291921 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.291941 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.292388 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.292757 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.293368 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.294688 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.295771 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.296388 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.296569 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.299279 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.299287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.299302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.299336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.301802 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.302385 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.302393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.302401 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.302563 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.303703 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.303710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.303712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.304380 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.304538 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.304940 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.305487 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":165,"threads_started":1}
I20260923 17:24:00.307821 14547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.295124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.23.214.188:43117)
0923 17:24:00.295937 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.295946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.295958 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:24:00.295990 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.295993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.296033 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.296120 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.297070 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.297859 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.299097 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.299446 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.299547 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.302774 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.302778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.302780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.302800 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.305315 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.305922 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.305925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.305927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.306007 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.306316 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.306319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.306321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.306588 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.306770 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.307516 (+ 746us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.307630 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":161,"threads_started":1}
I20260923 17:24:00.308121 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.297315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.23.214.190:37869)
0923 17:24:00.297476 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.297481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.297495 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:00.297535 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.297538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.297579 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.297658 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.298600 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.299200 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.300665 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.300906 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.301030 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.304161 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.304164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.304166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.304192 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.306218 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.306790 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.306794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.306796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.306888 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.307166 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.307170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.307172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.307444 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.307546 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.307860 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.307979 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:24:00.308791 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.296672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:37869 (local address 127.0.0.1:54518)
0923 17:24:00.297028 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.297081 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.297180 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.297739 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.297745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.298213 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.298373 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.298384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.299309 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.299312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.300339 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.300348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.301031 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.301035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.301142 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.302267 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.302286 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.304005 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.306348 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.306357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.306362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.306668 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.306997 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.306999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.307000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.307069 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.307546 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.307553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.307664 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.307805 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.307979 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260923 17:24:00.310022 14543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:00.311422 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: Bootstrap starting.
I20260923 17:24:00.314229 14467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:00.315138 14467 log.cc:824] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184240, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:24:00.317508 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: No bootstrap required, opened a new log
I20260923 17:24:00.319829 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.310586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.23.214.189:33017)
0923 17:24:00.310697 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.310700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.310845 (+ 145us) server_negotiation.cc:407] Connection header received
0923 17:24:00.311022 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.311025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.311072 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.311150 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.311582 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.312125 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.313008 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.313346 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.313751 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.316427 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.316430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.316433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.316457 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.317947 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.318373 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.318377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.318379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.318462 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.318761 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.318764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.318766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.319063 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.319187 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.319562 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.319682 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:24:00.319831 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.310530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:33017 (local address 127.0.0.1:59390)
0923 17:24:00.310765 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.310775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.310906 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.311180 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.311183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.311354 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.311454 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.311460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.312246 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.312248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.312889 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.312897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.313767 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.313773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.313919 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.314390 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.314409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.316273 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.318045 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.318050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.318053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.318255 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.318562 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.318565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.318567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.318648 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.319236 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.319241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.319370 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.319507 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.319648 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260923 17:24:00.320806 14543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:00.321089 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.309587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.23.214.190:37869)
0923 17:24:00.309691 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.309695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.309712 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:24:00.309745 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.309747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.309778 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.309864 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.310858 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.311589 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.312621 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.312965 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.313057 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.316550 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.316553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.316556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.316582 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.318792 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.319357 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.319360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.319362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.319463 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.319777 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.319781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.319782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.319994 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.320214 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.320820 (+ 606us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.320935 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:24:00.322001 14467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.322265 14467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f70be81a86544ab9c582381523c16da, State: Initialized, Role: FOLLOWER
I20260923 17:24:00.322642 14467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.323168 14550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.323313 14550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:00.324023 14467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184240, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:24:00.331331 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.321321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:58032)
0923 17:24:00.321675 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.321689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.321780 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.322178 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.322181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.322504 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.322611 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.322621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.323853 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.323856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.324690 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.324700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.325359 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.325362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.325434 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.326008 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.326027 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.327665 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.329654 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.329657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.329659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.329964 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.330252 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.330254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.330256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.330330 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.330836 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.330839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.330922 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.331015 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.331151 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260923 17:24:00.331808 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.321341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.23.214.188:43117)
0923 17:24:00.321439 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.321445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.321933 (+ 488us) server_negotiation.cc:407] Connection header received
0923 17:24:00.321963 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.321966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.322014 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.322082 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.322726 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.323742 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.324817 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.325158 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.325225 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.327815 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.327819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.327821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.327847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.329524 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.330069 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.330072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.330073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.330161 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.330433 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.330436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.330437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.330710 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.330805 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.331544 (+ 739us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.331666 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260923 17:24:00.331771 14561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:00.331844 14561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:00.334982 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.322193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.23.214.189:33017)
0923 17:24:00.322314 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.322320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.322365 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:24:00.322504 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.322507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.322546 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.322625 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.323387 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.324101 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.325151 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.325472 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.325555 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.329994 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.329998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.330001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.330027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.331814 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.333035 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.333042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.333046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.333159 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.333501 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.333504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.333506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.333733 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.333999 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.334688 (+ 689us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.334830 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:24:00.335299 14543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: Bootstrap starting.
I20260923 17:24:00.328047 14399 master_runner.cc:186] Error getting master registration for 127.23.214.190:37869: OK, instance_id { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" instance_seqno: 1790184239810957 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:24:00.336046 14399 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:24:00.338245 14543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:00.339238 14543 log.cc:824] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:00.339617 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:24:00.341526 14543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: No bootstrap required, opened a new log
I20260923 17:24:00.341089 14366 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "19041f41db804d16ae0ebf40d23356d4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530
I20260923 17:24:00.345849 14543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.346174 14543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0aa602adcd49e4a512a0cdb72ee459, State: Initialized, Role: FOLLOWER
I20260923 17:24:00.346553 14543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.347113 14564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.347282 14564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:00.348054 14543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:24:00.353328 14575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:00.353420 14575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:00.351965 14475 master_runner.cc:186] Error getting master registration for 127.23.214.190:37869: OK, instance_id { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" instance_seqno: 1790184239810957 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:24:00.355472 14475 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:24:00.447214 14392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } attempt: 4
Sep 23 17:24:00 dist-test-slave-9q7f krb5kdc[14311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184239, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
I20260923 17:24:00.458206 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.447987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.23.214.188:43117)
0923 17:24:00.448137 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.448142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.448232 (+ 90us) server_negotiation.cc:407] Connection header received
0923 17:24:00.448346 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.448349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.448405 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.448513 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.449027 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.449914 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.451279 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.451636 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.451744 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.454478 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.454482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.454485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.454515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.456281 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.456814 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.456818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.456820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.456902 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.457228 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.457231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.457233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.457456 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.457554 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.457911 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.458041 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:24:00.458217 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.447879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:43117 (local address 127.0.0.1:58036)
0923 17:24:00.448109 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:00.448126 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:00.448251 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:00.448525 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:00.448528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:00.448749 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:00.448873 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.448880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.450170 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.450174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.451112 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:00.451127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.451789 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.451793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:00.451878 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.452485 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:00.452503 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:00.454303 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:00.456429 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.456433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.456435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.456710 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.457015 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:00.457019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:00.457021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:00.457127 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:00.457581 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:00.457585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:00.457710 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 17:24:00.457830 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 17:24:00.458022 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260923 17:24:00.462275 14392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4: Bootstrap starting.
I20260923 17:24:00.465404 14392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:00.466456 14392 log.cc:824] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:00.468757 14392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4: No bootstrap required, opened a new log
I20260923 17:24:00.472097 14392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.472348 14392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19041f41db804d16ae0ebf40d23356d4, State: Initialized, Role: FOLLOWER
I20260923 17:24:00.472697 14392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.474174 14392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:00.474340 14366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:24:00.474502 14366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:00.475811 14366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.477504 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19041f41db804d16ae0ebf40d23356d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f70be81a86544ab9c582381523c16da"
I20260923 17:24:00.477756 14441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:00.479107 14441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19041f41db804d16ae0ebf40d23356d4 in term 1.
I20260923 17:24:00.479075 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.480157 14517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19041f41db804d16ae0ebf40d23356d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c0aa602adcd49e4a512a0cdb72ee459"
I20260923 17:24:00.480412 14517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:00.481703 14517 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19041f41db804d16ae0ebf40d23356d4 in term 1.
I20260923 17:24:00.479629 14366 leader_election.cc:288] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [CANDIDATE]: Term 1 election: Requested vote from peers 7f70be81a86544ab9c582381523c16da (127.23.214.189:33017), 3c0aa602adcd49e4a512a0cdb72ee459 (127.23.214.188:43117)
W20260923 17:24:00.483552 14589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:00.483702 14589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:00.482271 14343 leader_election.cc:302] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19041f41db804d16ae0ebf40d23356d4, 3c0aa602adcd49e4a512a0cdb72ee459; no voters:
I20260923 17:24:00.489005 14324 master_runner.cc:263] No leader master found from master 19041f41db804d16ae0ebf40d23356d4
I20260923 17:24:00.489096 14324 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:24:00.479195 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:00.489260 14578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:00.489354 14578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [term 1 LEADER]: Becoming Leader. State: Replica: 19041f41db804d16ae0ebf40d23356d4, State: Running, Role: LEADER
I20260923 17:24:00.489688 14578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } }
I20260923 17:24:00.492036 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19041f41db804d16ae0ebf40d23356d4. Latest consensus state: current_term: 1 leader_uuid: "19041f41db804d16ae0ebf40d23356d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.492184 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:00.495191 14597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:00.496218 14597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:00.511152 14517 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [term 1 FOLLOWER]: Refusing update from remote peer 19041f41db804d16ae0ebf40d23356d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:00.512014 14595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:00.512444 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.492789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.23.214.189:33017)
0923 17:24:00.492952 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.492958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.497303 (+ 4345us) server_negotiation.cc:407] Connection header received
0923 17:24:00.497917 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.497921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.497978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.498054 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.503038 (+ 4984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.503828 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.505741 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.506076 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.506163 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.508130 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.508134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.508138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.508169 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.510243 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.510791 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.510796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.510800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.510890 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.511173 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.511176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.511178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.511397 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.511502 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.511891 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.512022 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:24:00.515956 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.493606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.23.214.188:43117)
0923 17:24:00.493735 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.493740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.497533 (+ 3793us) server_negotiation.cc:407] Connection header received
0923 17:24:00.498830 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.498836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.498900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.498960 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.502026 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.502807 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.506718 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.507074 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.507174 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.511858 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.511862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.511864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.511893 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.514093 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.514593 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.514596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.514598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.514687 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.514990 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.514993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.514995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.515248 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.515358 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.515697 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.515814 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:24:00.516340 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:00.494453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.23.214.190:37869)
0923 17:24:00.494560 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.494564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.498093 (+ 3529us) server_negotiation.cc:407] Connection header received
0923 17:24:00.498288 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.498292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.498338 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.498393 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.502015 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.502803 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.507879 (+ 5076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:00.508209 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:00.508278 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:00.512246 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:00.512249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:00.512252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:00.512279 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:00.514524 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.515012 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.515016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.515018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.515109 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:00.515405 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:00.515408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:00.515410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:00.515616 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:00.515740 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:00.516069 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 17:24:00.516197 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 17:24:00.519948 14564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19041f41db804d16ae0ebf40d23356d4. Latest consensus state: current_term: 1 leader_uuid: "19041f41db804d16ae0ebf40d23356d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.520071 14564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:00.521227 14441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [term 1 FOLLOWER]: Refusing update from remote peer 19041f41db804d16ae0ebf40d23356d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:00.521863 14595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:00.526439 14550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19041f41db804d16ae0ebf40d23356d4. Latest consensus state: current_term: 1 leader_uuid: "19041f41db804d16ae0ebf40d23356d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.526592 14550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:24:00.528827 14496 tablet.cc:2369] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:00.529659 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19041f41db804d16ae0ebf40d23356d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.529791 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:00.530264 14601 mvcc.cc:205] Tried to move back new op lower bound from 7332594649124020224 to 7332594649050214400. Current Snapshot: MvccSnapshot[applied={T|T < 7332594649124020224}]
I20260923 17:24:00.530433 14564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19041f41db804d16ae0ebf40d23356d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.530512 14602 mvcc.cc:205] Tried to move back new op lower bound from 7332594649124020224 to 7332594649050214400. Current Snapshot: MvccSnapshot[applied={T|T < 7332594649124020224}]
I20260923 17:24:00.530557 14564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:24:00.530891 14420 tablet.cc:2369] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:00.532517 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19041f41db804d16ae0ebf40d23356d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.532653 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:00.533726 14550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19041f41db804d16ae0ebf40d23356d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19041f41db804d16ae0ebf40d23356d4" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 37869 } } peers { permanent_uuid: "7f70be81a86544ab9c582381523c16da" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 33017 } } peers { permanent_uuid: "3c0aa602adcd49e4a512a0cdb72ee459" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 43117 } } }
I20260923 17:24:00.533855 14550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:00.533813 14604 mvcc.cc:205] Tried to move back new op lower bound from 7332594649124020224 to 7332594649050214400. Current Snapshot: MvccSnapshot[applied={T|T < 7332594649124020224}]
I20260923 17:24:00.535442 14597 catalog_manager.cc:1347] Generated new cluster ID: b7bd0d2c13d64145a3b81885835c9e01
I20260923 17:24:00.535504 14597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:00.555135 14597 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:00.556535 14597 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:00.571050 14597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19041f41db804d16ae0ebf40d23356d4: Generated new TSK 0
I20260923 17:24:00.572062 14597 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:24:00.596416 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.594742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.23.214.189:33017)
0923 17:24:00.594890 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.594896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.595351 (+ 455us) server_negotiation.cc:407] Connection header received
0923 17:24:00.595547 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.595550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.595598 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.595649 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.596279 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59416: BlockingRecv error: recv got EOF from 127.0.0.1:59416 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 17:24:00.596421 14396 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.594783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.23.214.190:37869)
0923 17:24:00.594909 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.594913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.595226 (+ 313us) server_negotiation.cc:407] Connection header received
0923 17:24:00.595457 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.595461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.595518 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.595576 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.596274 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54548: BlockingRecv error: recv got EOF from 127.0.0.1:54548 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 17:24:00.597289 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.595427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.23.214.188:43117)
0923 17:24:00.595549 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.595554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.596025 (+ 471us) server_negotiation.cc:407] Connection header received
0923 17:24:00.596199 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.596203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.596257 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.596328 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.597167 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58060: BlockingRecv error: recv got EOF from 127.0.0.1:58060 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 17:24:00.600442 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.599309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.23.214.189:33017)
0923 17:24:00.599425 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.599429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.599548 (+ 119us) server_negotiation.cc:407] Connection header received
0923 17:24:00.599683 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.599686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.599762 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.599848 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.600332 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59428: BlockingRecv error: recv got EOF from 127.0.0.1:59428 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 17:24:00.604665 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.603814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.23.214.189:33017)
0923 17:24:00.603941 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.603944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.603955 (+ 11us) server_negotiation.cc:407] Connection header received
0923 17:24:00.603998 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.604000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.604039 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.604133 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.604532 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59442: BlockingRecv error: recv got EOF from 127.0.0.1:59442 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 17:24:00.609755 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.608497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.23.214.189:33017)
0923 17:24:00.608612 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.608616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.608946 (+ 330us) server_negotiation.cc:407] Connection header received
0923 17:24:00.609096 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.609099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.609150 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.609237 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.609646 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59452: BlockingRecv error: recv got EOF from 127.0.0.1:59452 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 17:24:00.616987 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.615760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.23.214.189:33017)
0923 17:24:00.615922 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.615929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.616048 (+ 119us) server_negotiation.cc:407] Connection header received
0923 17:24:00.616217 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.616220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.616275 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.616344 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.616874 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59456: BlockingRecv error: recv got EOF from 127.0.0.1:59456 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260923 17:24:00.628422 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.627300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.23.214.189:33017)
0923 17:24:00.627452 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.627457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.627603 (+ 146us) server_negotiation.cc:407] Connection header received
0923 17:24:00.627710 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.627714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.627773 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.627878 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.628307 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59460: BlockingRecv error: recv got EOF from 127.0.0.1:59460 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 17:24:00.647904 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.646692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.23.214.189:33017)
0923 17:24:00.646850 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.646854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.646969 (+ 115us) server_negotiation.cc:407] Connection header received
0923 17:24:00.647111 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.647114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.647161 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.647233 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.647757 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59474: BlockingRecv error: recv got EOF from 127.0.0.1:59474 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 17:24:00.683384 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.682257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.23.214.189:33017)
0923 17:24:00.682411 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.682415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.682585 (+ 170us) server_negotiation.cc:407] Connection header received
0923 17:24:00.682715 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.682718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.682774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.682861 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.683282 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59488: BlockingRecv error: recv got EOF from 127.0.0.1:59488 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:24:00.751477 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.750194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.23.214.189:33017)
0923 17:24:00.750406 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.750412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.750630 (+ 218us) server_negotiation.cc:407] Connection header received
0923 17:24:00.750734 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.750737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.750791 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.750874 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.751338 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59504: BlockingRecv error: recv got EOF from 127.0.0.1:59504 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 17:24:00.882860 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:00.881586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.23.214.189:33017)
0923 17:24:00.881737 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:00.881742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:00.881915 (+ 173us) server_negotiation.cc:407] Connection header received
0923 17:24:00.882109 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:00.882117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:00.882165 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:00.882261 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:00.882746 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59514: BlockingRecv error: recv got EOF from 127.0.0.1:59514 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 17:24:01.142179 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:01.141119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.23.214.189:33017)
0923 17:24:01.141272 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:01.141278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:01.141294 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:24:01.141450 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:01.141452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:01.141499 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:01.141569 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:01.142050 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59516: BlockingRecv error: recv got EOF from 127.0.0.1:59516 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:24:01.333303 14561 catalog_manager.cc:1259] Loaded cluster ID: b7bd0d2c13d64145a3b81885835c9e01
I20260923 17:24:01.333444 14561 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: loading cluster ID for follower catalog manager: success
I20260923 17:24:01.335534 14561 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: acquiring CA information for follower catalog manager: success
I20260923 17:24:01.336921 14561 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f70be81a86544ab9c582381523c16da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:24:01.354708 14575 catalog_manager.cc:1259] Loaded cluster ID: b7bd0d2c13d64145a3b81885835c9e01
I20260923 17:24:01.354794 14575 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: loading cluster ID for follower catalog manager: success
I20260923 17:24:01.356467 14575 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: acquiring CA information for follower catalog manager: success
I20260923 17:24:01.357676 14575 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c0aa602adcd49e4a512a0cdb72ee459: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:24:01.660427 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:01.656245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.23.214.189:33017)
0923 17:24:01.656529 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:01.656537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:01.656578 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:24:01.656782 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:01.656786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:01.656926 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:01.657058 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:01.657328 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.657901 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.658789 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.659030 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.659137 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:01.659544 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:01.660002 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:01.660143 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:01.660187 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 17:24:01.660248 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260923 17:24:01.667371 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:01.663626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.23.214.188:43117)
0923 17:24:01.663848 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:01.663852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:01.663865 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:24:01.663969 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:01.663971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:01.664092 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:01.664193 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:01.664419 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.664960 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.665838 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.666162 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.666254 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:01.666520 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:01.666959 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:01.667054 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:01.667099 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 17:24:01.667147 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260923 17:24:01.674526 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:01.670536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.23.214.189:33017)
0923 17:24:01.670791 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:01.670796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:01.670810 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:01.670961 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:01.670964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:01.671023 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:01.671105 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:01.671466 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.672317 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.673257 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.673496 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.673606 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:01.674034 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:01.674116 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:01.674197 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:01.674317 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 17:24:01.674375 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260923 17:24:01.675247 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:01.670956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.23.214.188:43117)
0923 17:24:01.671215 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:01.671219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:01.671445 (+ 226us) server_negotiation.cc:407] Connection header received
0923 17:24:01.671635 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:01.671641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:01.671702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:01.671764 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:01.671983 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.672546 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.673938 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.674259 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.674355 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:01.674779 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:01.674856 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:01.674903 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:01.675059 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 17:24:01.675108 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260923 17:24:01.675655 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:01.670525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.23.214.190:37869)
0923 17:24:01.670976 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:01.670980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:01.670994 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:01.671173 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:01.671180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:01.671347 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:01.671500 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:01.671721 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.672491 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.673948 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:01.674252 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:01.674701 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:01.674845 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:01.675246 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:01.675343 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:01.675445 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:24:01.675509 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":80,"threads_started":1}
I20260923 17:24:01.678573 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14324
I20260923 17:24:01.685004 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14399
I20260923 17:24:01.691182 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14475
2026-09-23T17:24:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3213 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:24:01 dist-test-slave-9q7f krb5kdc[14691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:24:01 dist-test-slave-9q7f krb5kdc[14691](info): set up 2 sockets
Sep 23 17:24:01 dist-test-slave-9q7f krb5kdc[14691](info): commencing operation
krb5kdc: starting...
W20260923 17:24:02.740491 24410 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/test-user.keytab.
Sep 23 17:24:02 dist-test-slave-9q7f krb5kdc[14691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:24:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:24:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.214.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
Entry for principal kudu/127.23.214.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab.
I20260923 17:24:02.867413 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45613
--webserver_interface=127.23.214.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:45613,127.23.214.189:36073,127.23.214.188:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:24:02.979120 14704 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:24:02.979486 14704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:02.979566 14704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:02.979650 14704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:02.983705 14704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:02.983798 14704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:02.983829 14704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:02.983855 14704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:02.983881 14704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:02.989200 14704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:45613,127.23.214.189:36073,127.23.214.188:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.190.keytab
--principal=kudu/127.23.214.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.190:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.214.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:02.990355 14704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:02.991425 14704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:03.000017 14712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:03.000061 14709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:03.000015 14710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:03.000495 14704 server_base.cc:1034] running on GCE node
I20260923 17:24:03.001027 14704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:03.002252 14704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:03.003461 14704 hybrid_clock.cc:648] HybridClock initialized: now 1790184243003447 us; error 25 us; skew 500 ppm
Sep 23 17:24:02 dist-test-slave-9q7f krb5kdc[14691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:03.006225 14704 init.cc:377] Logged in from keytab as kudu/127.23.214.190@KRBTEST.COM (short username kudu)
I20260923 17:24:03.007061 14704 webserver.cc:460] Webserver started at http://127.23.214.190:42473/ using document root <none> and password file <none>
I20260923 17:24:03.007576 14704 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:03.007639 14704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:03.007804 14704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:03.009799 14704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data/instance:
uuid: "568760260af2417e8a2ef2740f7df8d6"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "a9e105a5ef92278f108a7cfe598cb520"
server_key_iv: "3eaf4c20617547b1496c2fb764e165dd"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.010423 14704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/wal/instance:
uuid: "568760260af2417e8a2ef2740f7df8d6"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "a9e105a5ef92278f108a7cfe598cb520"
server_key_iv: "3eaf4c20617547b1496c2fb764e165dd"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.014258 14704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:24:03.016808 14719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:03.018155 14704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:24:03.018263 14704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/wal
uuid: "568760260af2417e8a2ef2740f7df8d6"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "a9e105a5ef92278f108a7cfe598cb520"
server_key_iv: "3eaf4c20617547b1496c2fb764e165dd"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.018354 14704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:03.039923 14704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:03.043229 14704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:03.043394 14704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:03.050328 14704 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.190:45613
I20260923 17:24:03.050616 14771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.190:45613 every 8 connection(s)
I20260923 17:24:03.051355 14704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/data/info.pb
I20260923 17:24:03.054157 14772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:03.057817 14772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:03.058524 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14704
I20260923 17:24:03.058660 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-0/wal/instance
I20260923 17:24:03.059000 24410 external_mini_cluster.cc:1428] Setting key 83cb2f8fc5b80da53aa056d473a69f0a
WARNING: no policy specified for kudu/127.23.214.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.189@KRBTEST.COM" created.
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:24:03.077608 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.061266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.23.214.190:45613)
0923 17:24:03.061531 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.061541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.061575 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:24:03.062556 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.062564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.062882 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.063084 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.063909 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.065460 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.066515 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.067155 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.067398 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.071961 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.071972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.071991 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.072027 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.074058 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.074733 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.074738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.074739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.074856 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.075318 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.075325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.075327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.075810 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.075930 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.076499 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.076750 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260923 17:24:03.077617 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.060424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45613 (local address 127.0.0.1:43896)
0923 17:24:03.061013 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.061074 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.062234 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.063086 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.063090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.063418 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.063743 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.063757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.065592 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.065594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.066392 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.066401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.067309 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.067311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.067434 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.068034 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.068050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.071771 (+ 3721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.074195 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.074207 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.074221 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.074550 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.074974 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.074980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.074985 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.075146 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.075949 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.075960 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.076158 (+ 198us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.076371 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.076869 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":101,"thread_start_us":133,"threads_started":1}
I20260923 17:24:03.080636 14772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:03.081497 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.081211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46452)
0923 17:24:03.081346 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.081429 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:24:03.081737 14721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
I20260923 17:24:03.082623 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.082404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46468)
0923 17:24:03.082516 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.082566 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:24:03.082785 14772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:36073: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab.
Entry for principal kudu/127.23.214.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab.
I20260923 17:24:03.088786 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:36073
--webserver_interface=127.23.214.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab
--principal=kudu/127.23.214.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:45613,127.23.214.189:36073,127.23.214.188:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:03.132069 14772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } attempt: 1
I20260923 17:24:03.132985 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.132624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46476)
0923 17:24:03.132810 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.132889 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:24:03.133721 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.133491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46478)
0923 17:24:03.133599 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.133652 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:24:03.133872 14772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:36073: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
W20260923 17:24:03.200845 14779 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:24:03.201121 14779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:03.201144 14779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:03.201234 14779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:24:03.202090 14772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } attempt: 2
I20260923 17:24:03.202929 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.202594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46488)
0923 17:24:03.202761 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.202838 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:24:03.203655 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.203440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46498)
0923 17:24:03.203542 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.203593 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 17:24:03.203802 14772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.189:36073: Network error: Client connection negotiation failed: client connection to 127.23.214.189:36073: connect: Connection refused (error 111)
W20260923 17:24:03.205807 14779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:03.205902 14779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:03.205951 14779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:03.205986 14779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:03.206018 14779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:03.211351 14779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:45613,127.23.214.189:36073,127.23.214.188:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.189.keytab
--principal=kudu/127.23.214.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.189:36073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.214.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:03.212495 14779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:03.213574 14779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:03.221992 14785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:03.222056 14787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:03.222002 14784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:03.222180 14779 server_base.cc:1034] running on GCE node
I20260923 17:24:03.222886 14779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:03.224038 14779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:03.225214 14779 hybrid_clock.cc:648] HybridClock initialized: now 1790184243225174 us; error 51 us; skew 500 ppm
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:03.228019 14779 init.cc:377] Logged in from keytab as kudu/127.23.214.189@KRBTEST.COM (short username kudu)
I20260923 17:24:03.228901 14779 webserver.cc:460] Webserver started at http://127.23.214.189:34479/ using document root <none> and password file <none>
I20260923 17:24:03.229463 14779 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:03.229524 14779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:03.229781 14779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:03.231828 14779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data/instance:
uuid: "be47632de576443d8535a58749296a93"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "a99174207cbbf9f18fc3d0d062b98f55"
server_key_iv: "44c3de6c97cb16dbc62d5e390c5dff86"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.232407 14779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/wal/instance:
uuid: "be47632de576443d8535a58749296a93"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "a99174207cbbf9f18fc3d0d062b98f55"
server_key_iv: "44c3de6c97cb16dbc62d5e390c5dff86"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.236119 14779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:24:03.238456 14794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:03.239652 14779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:24:03.239794 14779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/wal
uuid: "be47632de576443d8535a58749296a93"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "a99174207cbbf9f18fc3d0d062b98f55"
server_key_iv: "44c3de6c97cb16dbc62d5e390c5dff86"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.239909 14779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:03.252466 14779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:03.255601 14779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:03.255818 14779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:03.262894 14779 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.189:36073
I20260923 17:24:03.263176 14846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.189:36073 every 8 connection(s)
I20260923 17:24:03.263921 14779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/data/info.pb
I20260923 17:24:03.265152 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14779
I20260923 17:24:03.265296 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-1/wal/instance
I20260923 17:24:03.265579 24410 external_mini_cluster.cc:1428] Setting key 83bb5e0a5691d3dba5e9fafa4893a57f
I20260923 17:24:03.266848 14847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:03.270691 14847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.214.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.214.188@KRBTEST.COM" created.
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:24:03.284925 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.272657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.23.214.190:45613)
0923 17:24:03.272807 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.272811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.273837 (+ 1026us) server_negotiation.cc:407] Connection header received
0923 17:24:03.274862 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.274865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.274919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.274995 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.276288 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.276858 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.278342 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.278594 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.278679 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.281190 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.281200 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.281204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.281230 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.282784 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.283327 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.283332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.283334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.283420 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.283764 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.283769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.283770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.283966 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.284077 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.284595 (+ 518us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.284729 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:24:03.285785 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.273140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45613 (local address 127.0.0.1:43908)
0923 17:24:03.273678 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.273721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.274658 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.275159 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.275165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.275623 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.276122 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.276148 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.276992 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.276995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.278188 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.278202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.278752 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.278760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.278975 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.279598 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.279616 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.281012 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.282928 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.282934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.282949 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.283211 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.283522 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.283524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.283526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.283666 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.284077 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.284085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.284347 (+ 262us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.284540 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.284865 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":144,"threads_started":1}
I20260923 17:24:03.287266 14847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.214.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab.
Entry for principal kudu/127.23.214.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab.
I20260923 17:24:03.294271 24410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskADzyP8/build/debug/bin/kudu
/tmp/dist-test-taskADzyP8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:45837
--webserver_interface=127.23.214.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab
--principal=kudu/127.23.214.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.214.148:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.214.190:45613,127.23.214.189:36073,127.23.214.188:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:24:03.294974 14772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } attempt: 3
I20260923 17:24:03.297079 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.287795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46508)
0923 17:24:03.287948 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.287959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.288020 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.288551 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.288554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.288728 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.288799 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.288804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.289745 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.289751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.290342 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.290350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.290961 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.290965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.291196 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.291737 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.291756 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.293280 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.295276 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.295281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.295283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.295525 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.295820 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.295823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.295824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.295940 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.296590 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.296592 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.296656 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.296763 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.296893 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":72}
I20260923 17:24:03.297192 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.287900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.23.214.189:36073)
0923 17:24:03.288231 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.288239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.288257 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:24:03.288288 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.288291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.288435 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.288575 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.288904 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.289632 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.290470 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.290845 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.290965 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.293421 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.293431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.293442 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.293463 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.295148 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.295646 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.295651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.295653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.295740 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.296064 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.296070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.296074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.296421 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.296548 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.296865 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.296988 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":254,"thread_start_us":139,"threads_started":1}
I20260923 17:24:03.298861 14847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:03.299577 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.299351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41730)
0923 17:24:03.299441 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.299519 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 17:24:03.300129 14795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
I20260923 17:24:03.301175 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.300888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41744)
0923 17:24:03.301029 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.301096 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 17:24:03.301438 14847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:45837: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:24:03.306105 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.295424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46518)
0923 17:24:03.295571 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.295589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.295668 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.296483 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.296489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.296690 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.296846 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.296859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.297984 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.297991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.299507 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.299523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.300281 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.300285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.300412 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.300917 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.300933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.302338 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.304536 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.304539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.304540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.304732 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.305051 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.305053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.305054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.305140 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.305611 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.305614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.305698 (+ 84us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.305780 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.305982 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 17:24:03.306282 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.295470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.23.214.189:36073)
0923 17:24:03.296252 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.296262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.296275 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:24:03.296306 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.296309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.296346 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.296440 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.297001 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.297825 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.299678 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.300146 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.300271 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.302488 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.302494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.302499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.302529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.304412 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.304839 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.304845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.304849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.304950 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.305233 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.305238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.305242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.305506 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.305626 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.305897 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.306040 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":108,"threads_started":1}
I20260923 17:24:03.306995 14772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:03.307890 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.307569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41756)
0923 17:24:03.307760 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.307822 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 17:24:03.308769 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.308418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41768)
0923 17:24:03.308578 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.308700 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260923 17:24:03.308979 14772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:45837: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
I20260923 17:24:03.340158 14772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } attempt: 1
I20260923 17:24:03.341141 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.340775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41772)
0923 17:24:03.340965 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.341042 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 17:24:03.341995 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.341738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41774)
0923 17:24:03.341856 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.341927 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:24:03.342154 14772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:45837: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
I20260923 17:24:03.350733 14847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } attempt: 1
I20260923 17:24:03.351572 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.351258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41782)
0923 17:24:03.351418 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.351486 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:24:03.352324 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.352100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41794)
0923 17:24:03.352215 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.352266 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:24:03.352475 14847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:45837: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
W20260923 17:24:03.411623 14854 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:24:03.411904 14854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:24:03.411928 14854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:24:03.411965 14854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:24:03.415884 14854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:24:03.415943 14854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:24:03.415963 14854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:24:03.415979 14854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:24:03.415995 14854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:24:03.417346 14772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } attempt: 2
I20260923 17:24:03.418385 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.417982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41796)
0923 17:24:03.418218 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.418290 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260923 17:24:03.419137 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.418906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41812)
0923 17:24:03.419021 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.419075 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:24:03.419286 14772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:45837: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
I20260923 17:24:03.420634 14847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } attempt: 2
I20260923 17:24:03.421262 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.420997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41814)
0923 17:24:03.421131 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.421193 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:24:03.422001 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.421738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41822)
0923 17:24:03.421855 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.421934 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:24:03.422204 14847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.214.188:45837: Network error: Client connection negotiation failed: client connection to 127.23.214.188:45837: connect: Connection refused (error 111)
I20260923 17:24:03.423054 14854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.214.148:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.214.190:45613,127.23.214.189:36073,127.23.214.188:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/krb5kdc/kudu_127.23.214.188.keytab
--principal=kudu/127.23.214.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.214.188:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.214.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 16:09:16 UTC on ccycloud.quasar-aignqz.root.comops.site
build id 4667495
I20260923 17:24:03.424292 14854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:24:03.425427 14854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:24:03.434516 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:03.434516 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:24:03.434561 14860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:24:03.434894 14854 server_base.cc:1034] running on GCE node
I20260923 17:24:03.435742 14854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:24:03.437000 14854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:24:03.438208 14854 hybrid_clock.cc:648] HybridClock initialized: now 1790184243438182 us; error 43 us; skew 500 ppm
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:24:03.447732 14854 init.cc:377] Logged in from keytab as kudu/127.23.214.188@KRBTEST.COM (short username kudu)
I20260923 17:24:03.448868 14854 webserver.cc:460] Webserver started at http://127.23.214.188:35181/ using document root <none> and password file <none>
I20260923 17:24:03.449484 14854 fs_manager.cc:359] Metadata directory not provided
I20260923 17:24:03.449584 14854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:24:03.449841 14854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:24:03.452129 14854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data/instance:
uuid: "41900ce77bb343cd805e7b566f8d31ed"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "ff9dd1984a02cb6a544f09e782e4f9a0"
server_key_iv: "07cd16b7feb04fb9c0914b1419bb7dcf"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.452731 14854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/wal/instance:
uuid: "41900ce77bb343cd805e7b566f8d31ed"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "ff9dd1984a02cb6a544f09e782e4f9a0"
server_key_iv: "07cd16b7feb04fb9c0914b1419bb7dcf"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.456771 14854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 17:24:03.459420 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:24:03.460752 14854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:24:03.460916 14854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data,/tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/wal
uuid: "41900ce77bb343cd805e7b566f8d31ed"
format_stamp: "Formatted at 2026-09-23 17:24:03 on dist-test-slave-9q7f"
server_key: "ff9dd1984a02cb6a544f09e782e4f9a0"
server_key_iv: "07cd16b7feb04fb9c0914b1419bb7dcf"
server_key_version: "encryptionkey@0"
I20260923 17:24:03.461047 14854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:24:03.486624 14854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:24:03.489614 14854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:24:03.489836 14854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:24:03.496961 14854 rpc_server.cc:307] RPC server started. Bound to: 127.23.214.188:45837
I20260923 17:24:03.497263 14922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.214.188:45837 every 8 connection(s)
I20260923 17:24:03.497958 14854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/data/info.pb
I20260923 17:24:03.501121 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:24:03.502854 24410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskADzyP8/build/debug/bin/kudu as pid 14854
I20260923 17:24:03.502996 24410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskADzyP8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790183952943234-24410-0/minicluster-data/master-2/wal/instance
I20260923 17:24:03.503317 24410 external_mini_cluster.cc:1428] Setting key d5b7fbb26028e1407e6523cda8ced38a
I20260923 17:24:03.505689 14923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:24:03.513378 14847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } attempt: 3
I20260923 17:24:03.518469 14772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } attempt: 3
I20260923 17:24:03.519388 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.504785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.23.214.188:45837)
0923 17:24:03.505218 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.505233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.505268 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:24:03.505989 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.506008 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.506402 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.506741 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.507564 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.508887 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.509737 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.510449 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.510612 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.513231 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.513241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.513255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.513287 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.516613 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.517078 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.517085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.517089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.517173 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.517519 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.517523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.517528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.517798 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.518012 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.518248 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.518359 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":299,"spinlock_wait_cycles":768,"thread_start_us":137,"threads_started":1}
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.189@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
I20260923 17:24:03.521341 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.507752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.23.214.190:45613)
0923 17:24:03.507889 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.507894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.508141 (+ 247us) server_negotiation.cc:407] Connection header received
0923 17:24:03.508269 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.508272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.508323 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.508405 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.509032 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.509545 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.510585 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.510880 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.511008 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.513945 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.513948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.513950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.513973 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.515931 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.516636 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.516642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.516647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.516739 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.517151 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.517154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.517157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.517435 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.517535 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.521068 (+ 3533us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.521178 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:24:03.521610 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.507670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.190:45613 (local address 127.0.0.1:43922)
0923 17:24:03.508002 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.508050 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.508174 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.508456 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.508460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.508788 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.508904 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.508910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.509665 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.509668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.510461 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.510471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.511041 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.511045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.511147 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.511621 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.511636 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.513744 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.516048 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.516059 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.516070 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.516531 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.516847 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.516852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.516855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.517037 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.517532 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.517538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.517787 (+ 249us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.517990 (+ 203us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.518315 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260923 17:24:03.523555 14923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.190@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
I20260923 17:24:03.525830 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.513851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41836)
0923 17:24:03.514073 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.514089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.514175 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.514477 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.514480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.514676 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.514784 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.514790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.515516 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.515521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.516400 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.516414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.517020 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.517023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.517142 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.519309 (+ 2167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.519323 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.520874 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.524063 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.524067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.524069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.524347 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.524675 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.524678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.524738 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.524832 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.525306 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.525309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.525416 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.525503 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.525649 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 17:24:03.526741 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.513904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.23.214.188:45837)
0923 17:24:03.514234 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.514246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.514260 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:24:03.514304 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.514308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.514362 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.514450 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.514901 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.515404 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.516531 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.516898 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.517018 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.521008 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.521012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.521015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.521042 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.523937 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.524473 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.524477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.524479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.524566 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.524945 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.524948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.524950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.525189 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.526371 (+ 1182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.526453 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.526571 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.190@KRBTEST.COM
I20260923 17:24:03.528812 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.518860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41840)
0923 17:24:03.519010 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.519025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.519095 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.519608 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.519611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.519778 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.519900 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.519907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.521492 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.521495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.522309 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.522320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.522958 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.522965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.523072 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.523530 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.523542 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.525012 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.527100 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.527106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.527111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.527384 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.527726 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.527729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.527731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.527821 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.528284 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.528287 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.528424 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.528519 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.528651 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:24:03.529505 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.519084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.23.214.188:45837)
0923 17:24:03.519345 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.519354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.519372 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:24:03.519407 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.519411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.519463 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.520018 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.520551 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.521357 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.522459 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.522821 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.523306 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.525151 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.525154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.525157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.525180 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.526959 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.527519 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.527525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.527530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.527614 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.527955 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.527958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.527960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.528145 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.529083 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.529186 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.529303 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":118,"threads_started":1}
I20260923 17:24:03.530658 14847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: Bootstrap starting.
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:24:03.532588 14772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6: Bootstrap starting.
I20260923 17:24:03.532707 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.522084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.23.214.190:45613)
0923 17:24:03.522398 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.522401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.522441 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:24:03.522582 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.522586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.522672 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.523500 (+ 828us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.523591 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.524324 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.525467 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.525812 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.526299 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.528729 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.528733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.528735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.528757 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.530505 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.530982 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.530985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.530987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.531059 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.531376 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.531379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.531381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.531581 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.532355 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.532429 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.532535 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260923 17:24:03.533828 14847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:03.534803 14847 log.cc:824] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:03.535221 14772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:03.536012 14772 log.cc:824] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:03.536064 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.524284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.23.214.189:36073)
0923 17:24:03.524404 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.524410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.524624 (+ 214us) server_negotiation.cc:407] Connection header received
0923 17:24:03.524793 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.524797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.524839 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.524889 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.525405 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.526961 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.528352 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.528673 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.528869 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.531896 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.531901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.531904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.531930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.533810 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.534502 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.534506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.534508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.534602 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.534994 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.534997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.534999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.535281 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.535375 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.535822 (+ 447us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.535958 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:24:03.536536 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.524041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.189:36073 (local address 127.0.0.1:46532)
0923 17:24:03.524564 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.524577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.524691 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.525023 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.525025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.525181 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.525297 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.525304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.527094 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.527100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.528209 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.528220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.529335 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.529342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.529478 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.530262 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.530279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.531730 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.534102 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.534108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.534111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.534373 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.534749 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.534752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.534754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.534850 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.535484 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.535489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.535611 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.536284 (+ 673us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.536433 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461}
I20260923 17:24:03.538118 14772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6: No bootstrap required, opened a new log
I20260923 17:24:03.538152 14923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:24:03.539701 14847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: No bootstrap required, opened a new log
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.214.189@KRBTEST.COM
I20260923 17:24:03.542631 14772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
I20260923 17:24:03.542910 14772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568760260af2417e8a2ef2740f7df8d6, State: Initialized, Role: FOLLOWER
I20260923 17:24:03.543268 14772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
I20260923 17:24:03.543816 14932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.544106 14932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:03.544348 14847 raft_consensus.cc:348] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
I20260923 17:24:03.544569 14847 raft_consensus.cc:729] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be47632de576443d8535a58749296a93, State: Initialized, Role: FOLLOWER
Sep 23 17:24:03 dist-test-slave-9q7f krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184243, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.214.188@KRBTEST.COM for kudu/127.23.214.188@KRBTEST.COM
I20260923 17:24:03.544828 14772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:24:03.544886 14725 tablet.cc:2369] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:24:03.544904 14847 consensus_queue.cc:260] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
I20260923 17:24:03.546269 14847 sys_catalog.cc:564] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:03.547158 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.533849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.23.214.189:36073)
0923 17:24:03.533978 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.533984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.534060 (+ 76us) server_negotiation.cc:407] Connection header received
0923 17:24:03.534222 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.534225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.534267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.534335 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.535176 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.535961 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.537007 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.537354 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.537434 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.540440 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.540444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.540447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.540472 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.543351 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.545492 (+ 2141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.545496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.545498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.545592 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.545993 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.545997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.545998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.546235 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.546424 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.546857 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.546984 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:24:03.550001 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.538638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.214.188:45837 (local address 127.0.0.1:41850)
0923 17:24:03.538918 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 17:24:03.538933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:24:03.539014 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:24:03.539383 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:24:03.539386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:24:03.539552 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:24:03.539649 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.539656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.540829 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.540833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.541624 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:24:03.541634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.542422 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.542426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:24:03.542508 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.543505 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:24:03.543525 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:24:03.544952 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:24:03.548194 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.548200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.548202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.548478 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.548818 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:24:03.548821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:24:03.548823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:24:03.548918 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:24:03.549424 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:24:03.549427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:24:03.549520 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 17:24:03.549634 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:24:03.549771 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260923 17:24:03.551283 14933 sys_catalog.cc:455] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.551432 14933 sys_catalog.cc:458] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:03.550704 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.538825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.23.214.188:45837)
0923 17:24:03.539169 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.539174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.539186 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:24:03.539219 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.539222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.539264 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.539829 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.539969 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.540713 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.541772 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.542270 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.542430 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.545526 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.545530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.545533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.545560 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.548062 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.548598 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.548604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.548608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.548704 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.549028 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.549036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.549039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.549286 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.549386 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.550357 (+ 971us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.550527 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
W20260923 17:24:03.552191 14944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:03.552279 14944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:03.549955 14704 master_runner.cc:186] Error getting master registration for 127.23.214.189:36073: OK, instance_id { permanent_uuid: "be47632de576443d8535a58749296a93" instance_seqno: 1790184243260947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:24:03.553264 14704 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:24:03.554914 24410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:24:03.554961 14923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: Bootstrap starting.
I20260923 17:24:03.556175 14746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "568760260af2417e8a2ef2740f7df8d6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43926
I20260923 17:24:03.556402 14746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:24:03.556555 14746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:03.557621 14923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: Neither blocks nor log segments found. Creating new log.
I20260923 17:24:03.557948 14746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
W20260923 17:24:03.558115 14956 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:03.558179 14956 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:03.558508 14923 log.cc:824] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: Log is configured to *not* fsync() on all Append() calls
I20260923 17:24:03.559952 14821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "568760260af2417e8a2ef2740f7df8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "be47632de576443d8535a58749296a93"
I20260923 17:24:03.560173 14821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:03.560561 14923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: No bootstrap required, opened a new log
I20260923 17:24:03.561509 14821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568760260af2417e8a2ef2740f7df8d6 in term 1.
I20260923 17:24:03.562004 14721 leader_election.cc:302] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568760260af2417e8a2ef2740f7df8d6, be47632de576443d8535a58749296a93; no voters:
I20260923 17:24:03.558712 14779 master_runner.cc:186] Error getting master registration for 127.23.214.188:45837: OK, instance_id { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" instance_seqno: 1790184243494974 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:24:03.562325 14779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:24:03.562294 14746 leader_election.cc:288] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [CANDIDATE]: Term 1 election: Requested vote from peers be47632de576443d8535a58749296a93 (127.23.214.189:36073), 41900ce77bb343cd805e7b566f8d31ed (127.23.214.188:45837)
I20260923 17:24:03.565168 14932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:24:03.565415 14932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [term 1 LEADER]: Becoming Leader. State: Replica: 568760260af2417e8a2ef2740f7df8d6, State: Running, Role: LEADER
I20260923 17:24:03.565276 14923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
I20260923 17:24:03.565507 14923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41900ce77bb343cd805e7b566f8d31ed, State: Initialized, Role: FOLLOWER
I20260923 17:24:03.565764 14932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
I20260923 17:24:03.565860 14923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } }
I20260923 17:24:03.566437 14966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.566706 14966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:03.567268 14923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:24:03.567333 14897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "568760260af2417e8a2ef2740f7df8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41900ce77bb343cd805e7b566f8d31ed"
I20260923 17:24:03.567560 14897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:24:03.567695 14932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 568760260af2417e8a2ef2740f7df8d6. Latest consensus state: current_term: 1 leader_uuid: "568760260af2417e8a2ef2740f7df8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.567829 14932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:03.569034 14897 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568760260af2417e8a2ef2740f7df8d6 in term 1.
W20260923 17:24:03.572383 14979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:24:03.572471 14979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:24:03.572928 14968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:24:03.578498 14968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:24:03.585105 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.565008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.23.214.188:45837)
0923 17:24:03.565109 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.565114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.568267 (+ 3153us) server_negotiation.cc:407] Connection header received
0923 17:24:03.568468 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.568472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.568519 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.568570 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.569769 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.570538 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.571706 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.572070 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.572161 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.580797 (+ 8636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.580802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.580805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.580850 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.583045 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.583584 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.583599 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.583601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.583694 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.584028 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.584031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.584033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.584274 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.584392 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.584770 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.584909 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 17:24:03.595700 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.564702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.23.214.189:36073)
0923 17:24:03.564812 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.564817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.581449 (+ 16632us) server_negotiation.cc:407] Connection header received
0923 17:24:03.582039 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.582043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.582103 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.582161 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.586829 (+ 4668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.587636 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.588809 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.589174 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.589253 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.591314 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.591319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.591322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.591355 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.593293 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.593950 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.593954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.593956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.594065 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.594428 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.594434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.594436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.594722 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.594859 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.595286 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.595441 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 17:24:03.604821 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:03.577810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.23.214.190:45613)
0923 17:24:03.577965 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.577970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.585671 (+ 7701us) server_negotiation.cc:407] Connection header received
0923 17:24:03.585859 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.585862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.585953 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.586013 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.590847 (+ 4834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.592013 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.593600 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:03.594052 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:03.594164 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:03.598256 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:24:03.598261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:24:03.598265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:24:03.598299 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:24:03.602435 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.602999 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.603004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.603006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.603098 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:24:03.603422 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:24:03.603426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:24:03.603428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:24:03.603670 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:24:03.603785 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:03.604505 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 17:24:03.604630 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:24:03.608470 14897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [term 1 FOLLOWER]: Refusing update from remote peer 568760260af2417e8a2ef2740f7df8d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:03.609321 14965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:03.615959 14966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 568760260af2417e8a2ef2740f7df8d6. Latest consensus state: current_term: 1 leader_uuid: "568760260af2417e8a2ef2740f7df8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.616097 14966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:03.619798 14821 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [term 1 FOLLOWER]: Refusing update from remote peer 568760260af2417e8a2ef2740f7df8d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:24:03.620396 14965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:24:03.624729 14965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "568760260af2417e8a2ef2740f7df8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.624872 14965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:03.626622 14966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "568760260af2417e8a2ef2740f7df8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.626750 14966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:03.626948 14982 mvcc.cc:205] Tried to move back new op lower bound from 7332594661776629760 to 7332594661649375232. Current Snapshot: MvccSnapshot[applied={T|T < 7332594661776629760}]
I20260923 17:24:03.627090 14933 sys_catalog.cc:455] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 568760260af2417e8a2ef2740f7df8d6. Latest consensus state: current_term: 1 leader_uuid: "568760260af2417e8a2ef2740f7df8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.627197 14933 sys_catalog.cc:458] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:03.628504 14983 mvcc.cc:205] Tried to move back new op lower bound from 7332594661776629760 to 7332594661649375232. Current Snapshot: MvccSnapshot[applied={T|T < 7332594661776629760}]
I20260923 17:24:03.634366 14968 catalog_manager.cc:1347] Generated new cluster ID: a7a684c5f8304113b21917b7a43930a7
I20260923 17:24:03.634438 14968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:24:03.654403 14932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "568760260af2417e8a2ef2740f7df8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.654557 14932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6 [sys.catalog]: This master's current role is: LEADER
I20260923 17:24:03.655761 14933 sys_catalog.cc:455] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "568760260af2417e8a2ef2740f7df8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568760260af2417e8a2ef2740f7df8d6" member_type: VOTER last_known_addr { host: "127.23.214.190" port: 45613 } } peers { permanent_uuid: "be47632de576443d8535a58749296a93" member_type: VOTER last_known_addr { host: "127.23.214.189" port: 36073 } } peers { permanent_uuid: "41900ce77bb343cd805e7b566f8d31ed" member_type: VOTER last_known_addr { host: "127.23.214.188" port: 45837 } } }
I20260923 17:24:03.655891 14933 sys_catalog.cc:458] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:24:03.656421 14985 mvcc.cc:205] Tried to move back new op lower bound from 7332594661776629760 to 7332594661649375232. Current Snapshot: MvccSnapshot[applied={T|T < 7332594661776629760}]
I20260923 17:24:03.672219 14968 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:24:03.674364 14968 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:24:03.696791 14968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 568760260af2417e8a2ef2740f7df8d6: Generated new TSK 0
I20260923 17:24:03.697752 14968 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:24:03.736589 14776 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.733320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.23.214.190:45613)
0923 17:24:03.733464 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.733469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.735296 (+ 1827us) server_negotiation.cc:407] Connection header received
0923 17:24:03.735566 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.735572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.735640 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.735705 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.736430 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43930: BlockingRecv error: recv got EOF from 127.0.0.1:43930 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 17:24:03.738930 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.734241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.23.214.189:36073)
0923 17:24:03.734365 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.734371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.737397 (+ 3026us) server_negotiation.cc:407] Connection header received
0923 17:24:03.737606 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.737609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.737670 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.737735 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.738789 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46570: BlockingRecv error: recv got EOF from 127.0.0.1:46570 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 17:24:03.740702 14929 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.734920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.23.214.188:45837)
0923 17:24:03.735055 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.735062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.738395 (+ 3333us) server_negotiation.cc:407] Connection header received
0923 17:24:03.739825 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.739829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.739889 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.739954 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.740576 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41880: BlockingRecv error: recv got EOF from 127.0.0.1:41880 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 17:24:03.746407 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.744376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.23.214.189:36073)
0923 17:24:03.744589 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.744596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.745203 (+ 607us) server_negotiation.cc:407] Connection header received
0923 17:24:03.745455 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.745460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.745555 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.745618 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.746282 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46576: BlockingRecv error: recv got EOF from 127.0.0.1:46576 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260923 17:24:03.752354 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.750536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.23.214.189:36073)
0923 17:24:03.750657 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.750662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.751293 (+ 631us) server_negotiation.cc:407] Connection header received
0923 17:24:03.751520 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.751524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.751584 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.751647 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.752241 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46586: BlockingRecv error: recv got EOF from 127.0.0.1:46586 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 17:24:03.759701 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.756965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.23.214.189:36073)
0923 17:24:03.757078 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.757083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.758579 (+ 1496us) server_negotiation.cc:407] Connection header received
0923 17:24:03.758805 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.758809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.758867 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.758956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.759573 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46602: BlockingRecv error: recv got EOF from 127.0.0.1:46602 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 17:24:03.768419 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.766643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.23.214.189:36073)
0923 17:24:03.766796 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.766801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.767367 (+ 566us) server_negotiation.cc:407] Connection header received
0923 17:24:03.767574 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.767577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.767636 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.767698 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.768301 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46616: BlockingRecv error: recv got EOF from 127.0.0.1:46616 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 17:24:03.781009 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.779956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.23.214.189:36073)
0923 17:24:03.780119 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.780124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.780139 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:24:03.780194 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.780197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.780247 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.780329 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.780893 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46618: BlockingRecv error: recv got EOF from 127.0.0.1:46618 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 17:24:03.800637 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.799488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.23.214.189:36073)
0923 17:24:03.799616 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.799621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.799831 (+ 210us) server_negotiation.cc:407] Connection header received
0923 17:24:03.799951 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.799953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.800002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.800086 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.800542 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46634: BlockingRecv error: recv got EOF from 127.0.0.1:46634 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 17:24:03.836272 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.834849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.23.214.189:36073)
0923 17:24:03.835018 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.835023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.835322 (+ 299us) server_negotiation.cc:407] Connection header received
0923 17:24:03.835396 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.835399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.835464 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.835561 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.836142 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46638: BlockingRecv error: recv got EOF from 127.0.0.1:46638 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 17:24:03.903716 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:03.902565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.23.214.189:36073)
0923 17:24:03.902720 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:03.902725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:03.902926 (+ 201us) server_negotiation.cc:407] Connection header received
0923 17:24:03.903032 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:03.903034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:03.903080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:03.903155 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:03.903597 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46648: BlockingRecv error: recv got EOF from 127.0.0.1:46648 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 17:24:04.035037 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:04.033988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.23.214.189:36073)
0923 17:24:04.034115 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:04.034119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:04.034192 (+ 73us) server_negotiation.cc:407] Connection header received
0923 17:24:04.034320 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:04.034322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:04.034370 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:04.034452 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:04.034903 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46656: BlockingRecv error: recv got EOF from 127.0.0.1:46656 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 17:24:04.294304 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:24:04.293168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.23.214.189:36073)
0923 17:24:04.293321 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:04.293329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:04.293366 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:24:04.293548 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:04.293555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:04.293604 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:04.293698 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:24:04.294198 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46670: BlockingRecv error: recv got EOF from 127.0.0.1:46670 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:24:04.559556 14956 catalog_manager.cc:1259] Loaded cluster ID: a7a684c5f8304113b21917b7a43930a7
I20260923 17:24:04.559649 14956 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: loading cluster ID for follower catalog manager: success
I20260923 17:24:04.561385 14956 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: acquiring CA information for follower catalog manager: success
I20260923 17:24:04.562631 14956 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be47632de576443d8535a58749296a93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:24:04.573706 14979 catalog_manager.cc:1259] Loaded cluster ID: a7a684c5f8304113b21917b7a43930a7
I20260923 17:24:04.573791 14979 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: loading cluster ID for follower catalog manager: success
I20260923 17:24:04.575522 14979 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: acquiring CA information for follower catalog manager: success
I20260923 17:24:04.576604 14979 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41900ce77bb343cd805e7b566f8d31ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:24:04.812326 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:04.808615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.23.214.189:36073)
0923 17:24:04.808867 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:04.808871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:04.808886 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:24:04.809005 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:04.809007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:04.809139 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:04.809237 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:04.809468 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.810011 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.810901 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.811129 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.811208 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:04.811473 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:04.811910 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:04.811996 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:04.812078 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 17:24:04.812125 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
I20260923 17:24:04.819213 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:04.815604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.23.214.188:45837)
0923 17:24:04.815850 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:04.815854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:04.815881 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:24:04.815955 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:04.815957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:04.816067 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:04.816162 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:04.816340 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.816859 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.817624 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.817860 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.818008 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:04.818346 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:04.818784 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:04.818888 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:04.818960 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 17:24:04.819009 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20260923 17:24:04.827241 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:04.822668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.23.214.188:45837)
0923 17:24:04.822797 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:04.822802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:04.823172 (+ 370us) server_negotiation.cc:407] Connection header received
0923 17:24:04.823302 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:04.823305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:04.823349 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:04.823427 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:04.823680 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.824512 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.825983 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.826197 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.826253 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:04.826693 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:04.826775 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:04.826856 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:04.827026 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 17:24:04.827092 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 17:24:04.827280 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:04.822631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.23.214.189:36073)
0923 17:24:04.822764 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:04.822769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:04.823219 (+ 450us) server_negotiation.cc:407] Connection header received
0923 17:24:04.823379 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:04.823382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:04.823437 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:04.823502 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:04.823845 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.824614 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.825958 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.826312 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.826421 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:04.826739 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:04.826810 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:04.826906 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:04.827065 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 17:24:04.827122 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:24:04.828429 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:24:04.822543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.23.214.190:45613)
0923 17:24:04.822831 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0923 17:24:04.822836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:24:04.823110 (+ 274us) server_negotiation.cc:407] Connection header received
0923 17:24:04.823240 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:24:04.823243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:24:04.823445 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:24:04.823562 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:24:04.824872 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.825394 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.826587 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:24:04.827004 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:24:04.827100 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:24:04.827546 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:24:04.827901 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:24:04.828004 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:24:04.828240 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 17:24:04.828312 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":61,"threads_started":1}
I20260923 17:24:04.831447 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14704
I20260923 17:24:04.838699 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14779
I20260923 17:24:04.844637 24410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskADzyP8/build/debug/bin/kudu with pid 14854
2026-09-23T17:24:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3154 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6367 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (291895 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST